Before trying to get the osticket mailing system working, you need to make sure that PHP's mail function is correctly configured.
Have you tried a simple mail script to make sure that the php mail() function is working? Replace recipient@abc.com with your email address.
test.php
<?php
if(mail("recipient@abc.com","subject","body","From: from@abc.com"))
echo "email sent";
else
echo "email failed";
?>
Upload it to your server and open up the file through a browser. Do you get an email?