That is controlled in several different places. in osTicket you can change it in:
Admin panel -> Settings -> Tickets -> Maximum File Size
Not that the maximum allowed here is controlled by the servers php.ini. To increase it beyond whats allowed in osTicket you have to do one of the following:
1. edit the following directives in your php.ini
upload_max_filesize = 10M
post_max_size = 10M
NOTE: some web hosts allow you to use a custom php.ini.
2. edit the following directives in your .htaccess
php_value upload_max_filesize 10M
php_value post_max_size 10M
NOTE: this only works if you are using Apache and you may still have to set this in the php.ini also. I'm not sure.