I've spent days working on this with Filemaker tech support and am still working on it. SSL is a requirement for me and the cert needs to match my domain so the FMI cert was not an option and caused a significant client disruption during the transition.
On a Mac you need to add sudo to the beginning of the command line and also the path to where the cert you got from Symantec is sitting on your server. I got the same error because it did not have write permission to create the serverCustom.pem in the CStore folder.
However, be careful because it did work for me and installed my cert but then immediately blocked all access to my database through WebDirect as well as through FMP clients. One step forward and 100 back.
After a stressful and not fun period of trying to troubleshoot with the entire production database down. I dragged out the serverCustom.pem and restarted the server. My databases instantly came back up. Now according to the documentation once you drag out that file it should revert back to the default FMI cert. For reasons neither I nor FMP tech support can explain, that did not happen for me. My cert remained in place and my databases became accessible.
It runs in direct conflict with some guidance in the manual but the end results is you should be up with your certs in place.
1 of 1 people found this helpful
I figured it out. It was my mistake. Here's how I did it for those of you having trouble installing SSL Certificate...
First of all my SSL Certificate provider is Symantac (former Verisign) so if you have a different provider, the steps might be a little different.
1. Server.app not installed. (Didn't know about this until I installed FMS 13)
2. Create a certificate request by typing "sudo fmsadmin certificate create "/CN=common name/O=organization name/C=US/ST=California" on the terminal. Type your password and press enter.
3. This will generate 2 files in /Library/FileMaker Server/CStore, serverKey.pem (private key) and serverRequest.pem (CSR)
4. Revoke and Replace my certificate at Symantec.com and submitted the serverRequest.pem
5. After 30 mins my certificate is ready. Symantec gave me 3 certificates. Primary Intermediate (saved as primary.cer), Secondary Intermediate (saved as secondary.cer) and End Key (saved as certificate.cer).
6. Double-click primary.cer and secondary.cer and Keychain will install them.
7. Type "sudo fmsadmin certificate import /path/certificate.cer" type your password and press enter.
8. FMS generated serverCustom.pem inside the CStore folder which is the combination of Certificate and Private Key.
9. Restarted the server and tested it. It worked.
I was getting the error because I wasn't paying attention, I was importing the primary.cer instead of the certificate.cer.
Hope this helps.