I had the same problem with a custom certificate. I found out that you need to have the certificate in .pem format for it to be imported by FileMaker Server. See help.filemaker.com/app/answers/detail/a_id/14174 for more info. Looks like Keychain Access can export your certificate in .pem format directly, so you won't need to convert it by other means.
Hope that helps!
Thank you for your post.
Joseph Parent UVM is correct (thanks for the post) you would need to follow FileMaker's instruction to import the certificate in which case you get a "servercustomer.pem" which FileMaker Server handles.
For more information, and step by step instruction, see page 70 "Using the CLI certificate command" of the FileMaker Server 14 Getting Started Guide.
As you mentioned it, FileMaker Server 14 is not compatible with OS X Server. For System Requirements click here.
I hope this information is of use.
Good for you I just did this today.
I also have GoDaddy and OS X Server.
The problem is that the P12 file contains the CER and the PEM file. You need to extract the PEM file using the command line.
Open your command line Terminal and enter the following command (replace both file paths and file names):
openssl pkcs12 -in /Users/path/to/your/file/CertificateExported.p12 -out Users/path/to/your/file/CertificateExported.pem -nodes
Then it will ask you for the file password. The resulting CertificateExported.pem can be imported in FMS with your CER file.
The resulting PEM file will be unencrypted. Make sure you delete the file after import.
FMS and OS X work very well together.