I like to run optimize every week, the tables get a bit of overhead, especially if you're storing attachments in the database, and doubly so if you have a large volume of tickets.
Otherwise it seems to like keeping old data around, the logs and message headers etc can get large, but not overly. We've over half a million tickets and had to move to filesystem storage for attachments, just for performance, now the whole database can fit in ram again and it's happy.
I have a plugin to archive old tickets, it's still in beta, but you can help test it if you want.
https://github.com/clonemeagain/osticket-plugin-archiver