Thank you for the post.
Chapter 5 of the FileMaker Server 12 Getting Started Guide is titled "Upgrading or moving an existing installation."
The guide suggests to use the original installer to uninstall; however, it sounds like that's what is not working.
If you stop the FileMaker processes in Activity Monitor, trash the entire FileMaker Server folder located at /Library/FileMaker Server and restart the computer, then that should also uninstall the program. Please make sure to first back up any databases, schedules, and groups as trashing the entire folder will remove those.
For more information on the location of the databases, backups, schedules, and groups see page 56 of the FileMaker Server 12 Getting Started Guide.
i figured that would work but since the documentation says what it says I wanted to make sure there wasn't something special the uninstaller did.