FileMaker Server comes with a self signed SSL certificate. It will work just fine. The only problem is that it is self signed and not authenticated by a certificate authority. What this means is that whenever you access this database with a web browser, you will get one of those warnings that site might not be safe, etc. If it is just you and your company using it, no problem. If the public may need to access it, then such warnings scare them off and you'll need an authenticated certificate.
A good explanation of things is at: http://www.filemaker.com/help/13/fms/en/index.html#page/fms/fmsh_cmdref.19.05.html
What this explanation leaves out is how to get a certificate. FileMaker does not support many certificates because it has to preload them on the iOS devices. So there are very limited number of ones and most are expensive in the $50+. But the Comodo ones are cheaper. However, a more expensive one may technically be more secure or better authenticated. You have to go to a certificate authority's web page to get the certificate and it will involve sending files back and forth to get an authenticated certificate that goes in the CStore folder in FileMaker Server via the "fmsadmin certificate import" function.
If you're cheap and want to go the inexpensive route, check out the discussion at: https://fmdev.filemaker.com/message/159772#159772
That's a good explanation.
Then, there is nothing to do on the Server configuration, only the the fmsadmin CERTFICATE command :
fmsadmin CERTIFICATE CREATE server_name ; by the way, I don't have a server name, I am using the ip address, but If I want to have one, I should
create my DNS, right ?
I remember, I can't have Web Services on my server with Filemaker Server working together at the same server machine, but DNS is one of the things
I can have, right ?
I'm pretty sure certificates only come as part of a domain name. So, yes, you will need to get a domain name for your server. You do not have to run your own DNS, but you can if you want. The DNS is just a database that aligns IPs and domains.
FileMaker does not officially support Mac OS X Server. The main reason is that WebDirect is programmed to take over and manage Apache web services. This is conflict with Apple's Server.app that normally manages Apache. Of course all management can be done at the command line if you are an OS expert and you don't need these configuration apps. But few FileMaker developers are at that level of the OS.
While not officially supported, if you have Mac OS X Server and run the Server.app, as long as you don't try to manage web services with that app (e.g., turn on web services, set up domains or virtual domains, etc.), then the other services seem to work just fine with FileMaker Server. Just keep in mind if you start having problems and have to phone FileMaker, that they will not assist you until you have a clean machine meeting their official technical specifications.
FileMaker Server used to work just fine with Apple's Server.app prior to WebDirect and I think it is a travisty that FileMaker does not support that anymore. One reason why is if you have a small client, that has plenty of capacity on the server, then the server should be able to run more services (e.g., DNS, mail, file sharing, etc.). I understand FileMaker runs best with nothing else competing, but FileMaker should not require that as a server service, that it can't operate with any other running services. In the Windows Server world, server admin's just laugh at you if you tell them there is service you want to run and it is not to allow any other services to run on the server. Most all servers in the world are running multiple services. It is only FileMaker that is making this requirement in the server service realm.
I think you will find the FileMaker certificate creation involves more steps than you think. It is not just a matter of creating a certificate. What you are doing is creating a file that you then have to submit to your certificate authority who then has to approve your account and that certificate and then after all of that, send you a file back that you import the FMSADMIN CERTIFICATE IMPORT function. Each of those steps has requirements from setting permissions to following the certificate authority's rules for creating a certificate.
We will help you through it, but make sure it is something you need to do. As far as I am concerned, it is the most complicated part of the FileMaker Server configuration.
Oh, I have assumed all along you have a static public WAN IP for your domain.