I run into a problem when our Terena certs didn't match what Filemaker inc had on the list of supported SSL certificate vendors.
After much testing i managed to get it working.
In short I copied in the certificate chain after I imported the certificate.
Do as follows
1.>sudo fmsadmin CERTIFICATE CREATE fms.x.x
Two files wil be created in the CStore folder.
Send the serverRequest.pem to your certificate provider.
When you get the certificate from your provider se to that you also download the chain file/s
2.>sudo fmsadmin CERTIFICATE IMPORT /downloaded/cert.pem
to be sure stop the filemaker server
3.>sudo launchctl stop com.filemaker.fms
backup the file first
4.>sudo cp /Library/FileMaker Server/CStore/serverCustom.pem /Library/FileMaker Server/CStore/serverCustom.pem.old
edit the file serverCustom.pem and copy in the chain code
5. sudo cat /Library/FileMaker Server/CStore/serverCustom.pem /downloaded/chain.pem > /Library/FileMaker Server/CStore/serverCustom.pem
Start the server again
6.>sudo launchctl start com.filemaker.fms
Hope I helped someone