So if I get your right, your OST install isn't sending email like it should. This could be for a variety of reasons.
1. in OST goto Admin panel -> Settings and make sure that Autoresponders (Global Setting) are set correctly. (New Ticket set to enable)
2. Check your PHP logs and see if there is an error in them.
(If you look at your php.ini and look for "error_log =" directive it should speed up your finding where your logs are located.)
3. Additionally it may help to test if PHP can actually send mail. Try creating a file called test.php, put it in your web tree. (Make sure you change recipient@example.com to your email address). Use your web browser to hit the file.
<?php
$to = "recipient@example.com";
$subject = "TEST";
$body = "Hi,\n\nDid you get this??";
if (mail($to, $subject, $body)) {
echo("<p>Message successfully sent!</p>");
} else {
echo("<p>Message delivery failed...</p>");
}
?>
Please let us know the results.