If you use https:// you will be required to setup an SSL certificate. Self-signed certificates will not work.
FileMaker provides a list of supported SSL certificate vendors:
Chapter 7 of the getting started guide explains the CSR generation request which you will need to get a certificate:
For regular http:// the trust/authentication message should not be there. It might be popping up due to your DNS redirect.