What version of server?
A recent Java update broke server console for all versions of server except the most recent bug fix release of FileMaker Server 11.
For More Information see: Unable to launch the application : FileMaker Server 11 Admin Console
This is one of many acknowledged bugs that can be found in the Known Bug List here in the Report an Issue section of the forum.
It can also be downloaded as a database file from: http://www.4shared.com/file/8orL8apk/FMP_Bugs.html