have you tried a simple php send email script to test?
<?
$to = "somebody@mydomain.com";
$from_header = "From: administrator@mydomain.com";
$subject = "Test Email";
$contents = "Test PHP Email";
mail($to, $subject, $contents, $from_header);
?>
Change the $to and $from_header to email address within your domain.
If your email is setup correctly, this will trigger email. If this works, then something likely in osticket setup is incorrect.