This is quite possibly related to FMS 16v3 not able to import DigiCert's intermediate certificate
I upgraded to FileMaker Server 16 this morning, and used this as an opportunity to purchase a new SSL cert from GeoTrust. I chose GeoTrust because they were recommended by FileMaker here, and because their pricing, while not the lowest, was tolerable (the lowest price option seemed a bit sketchy to me).
Once I had purchased the cert and had gone through the (appropriate) verification process, I downloaded the cert and was ready to install. The instructions from FileMaker here under Purchase a SSL Certificate did not quite match what I saw when downloading the GeoTrust cert. (I chose "other" for the platform, as instructed, but had to select between "PKCS#7" and "X509" as my two choices for the certificate format. It turns out X509 more closely resembles what the FileMaker instructions suggest, although even that wasn't a perfect match. FileMaker said the cert file would be called yourDomainName.crt, but it was called ssl_certificate.cer.)
The FileMaker instructions say I must "concatenate the root and intermediate certificates into a single file" and call the concatenated file chain.pem. I didn't receive a root file, so I tried using my ssl_certificate.cer file and concatenating that with the IntermediateCA.cer file.
To make a long story short, in the various configurations I tried, I was able to import the cert into FileMaker Server, but the certificate is not recognized. When I use a FileMaker Client on a client computer to attach to files on the Server, I can access them, but they do not have a secure connection. When I use WebDirect coming in through the Internet, the files cannot be accessed at all.
In frustration, I connected with GeoTrust and a very helpful gentleman named Quinton and I worked for about an hour and a half trying various scenarios, but without success. Here is a summary of what we tried.
|Signed Certificate File||Private Key File||Intermediate Certificate File (concatenated components are listed in sequence)||Results/Notes|
|ssl_certificate.cer||serverKey.pem||chain.pem (ssl_certificate/IntermediateCA)||Since I did not have a root file, I (wrongly) assumed I should use the ssl_certificate.cer file. DID NOT LOAD.|
|ssl_certificate.cer||serverKey.pem||IntermediateCA.cer||Cert loaded. But FMP connections insecure and Web Direct inaccessible.|
GeoTrust consultant suggested renaming signed certificate file with .pem file type.
Cert loaded, FMP connections insecure, Web Direct inaccessible.
|ssl_certificate.pem||serverKey.pem||(no intermediate cert at all)||Cert loaded, FMP connections insecure, WebDirect inaccessible. We also tried using TSGal's suggested terminal command "fmsadmin certificate delete" to make sure the intermediate cert was cleared. It was, but the serverKey.pem file was also deleted. Fortunately we were able to recover it from the FMS_Removed folder inside the CStore folder.|
|ssl_certificate.pem||serverKey.pem||chain.pem (root/intermediate)||Since GeoTrust had not sent the root cert suggested by FileMaker, the GeoTrust consultant created one and emailed it to me. We built the concatenated chain.pem file as suggested, but this DID NOT LOAD.|
|ssl_certificate.pem||serverKey.pem||chain.pem (intermediate/root)||The GeoTrust consultant suggested that the root really should be at the bottom, so we tried concatenating the chain.pem file in a different sequence, but this also DID NOT LOAD.|
So right now I have purchased an SSL cert that does not secure the connection and that has broken Web Direct access completely. My first job will be to try to restore things to the way they were.
For the record, the way I know that the connection between FileMaker Pro clients and the Server is that the "lock" icon is orange with an X, not green. The way I know Web Direct access is completely down is that browsers get a message "can't establish a secure connection". And the message I get when trying to unsuccessfully load a cert combination (as in the table above) is "Certificate could not be imported: Config_DBServer_CertificateDialog_ErrorGeoTrust RSA CA 2018"
But both Quinton of GeoTrust and I want to figure this out, so we're asking the FileMaker community for any thoughts they have. Meanwhile, I'm going to try to uninstall the cert for now to get things back to the way they were pre-cert.