Thanks for posting!
This error message is a bit generic so let's start with an uninstall and reinstall. Follow the instructions below and let us know if the issue persists.
1. Close all of the files through the Databases view in the admin console.
2. Stop the Database Server through the admin console.
3. Move the /Library/FileMaker Server/Data folder to a safe place. This contains all of your databases, back ups, system level scripts, etc.
4. Run the installer for FileMaker Server 11 and opt to uninstall when prompted.
5. Move the /Library/FileMaker Server folder to the trash. This contains some configuration files that may cause a conflict on reinstall.
7. Run the installer to reinstall.
Ok, I tried to do an reinstall. The problem I run into now is that its not a install it an update. When i run the update I get "can't find a valid 11.0.1 or 11.0.2 version". Should just go back to 11.0.1 and not try to run version 11.0.3? If I do this my admin interface doesn't work because of java incompatibilities.
Let's try redownloading the update and attempting the installation again:
If it still fails after a redownload, let us know and we'll proceed from there.