One of the script steps, "Send Mail" has configuration options for the SMTP server that you want to use to send the mail. However, I think you can only use this if you "compose" the email inside FileMaker solution. That is, I don't think that you can have FM open an email client to create the email, and then send it via the server's SMTP service. Unless, you configure you email client to use the server's SMTP yourself.
To configure the "Send Mail" script step:
* Add the step to your script; click on the "Specify" button in the lower right.
* In the resulting window, at the top there is a drop down that probably says "Email Client"; change that to "SMTP server". When you do that, a button will show up that says "Configure".
* Click the Configure button to bring up the SMTP configuration options.
It is also possible to do all this from the FMServer.app utility. I don't have that open at the moment to give you steps on that, though.
Hope this helps some.