try different scenarios
Im having the same issue its not remotely close to the PHP or Server those are irrelevant.
people on this forum are complaining about the same problems when using a windows server. So PHP and server can be relevant.
Trying to find out what's happening:
* dit you select within settings - preferences all the bullets for sending out mail? Try selecting every 'enable' button for sending mail as well as checking all the checkboxes at Alerts & Notices. That's probably too much for your live installation, but it will help you with testing everything
* walk trough different scenario's: creating ticket by sending an e-mail, creating ticket using the customer frontend (http://www.domain.com/support/open.php(http://www.domain.com/support/open.php)), creating ticket using the backend (http://www.domain.com/support/scp/tickets.php?a=open(http://www.domain.com/support/scp/tickets.php?a=open)), sending reply by e-mail, sending reply by frontend, sending reply by backend. Assign a ticket to a staffmember and do the replies again. It's important to use different subjects, so you can find out which ticket matches which scenario. Use different e-mailaddresses for admin, staffmember, customer.
If no e-mails are send you do have a general e-mailproblem. If some of the e-mails are send, the problem is probably related to some osticket code.