The files created by the fmsadmin certificate commands are placed in the CStore folder under your FileMaker Server install folder:
<drive letter>:\Program Files\FileMaker\FileMaker Server\CStore on Windows, not really sure where the same location is on a Mac.
The files you want to look for are serverKey.pem and serverCustom.pem.
serverKey.pem is the private key and serverCustom.pem is the private key combined with the SSL certificate. This is the 'certificate' file that FileMaker is using. I don't see any reason why you couldn't just delete these files, especially if you have an invalid certificate.
I'm not 100% sure if running the fmsadmin certificate commands will overwrite the old files.
Thanks! We were able to delete those files yesterday and successfully install an SSL certificate