The problem is Java. Have you made sure you have the latest version of Java. You can also reset Java if it is giving you a hard time. I have a server that is still not running Java 7 properly and won't open Admin console, but I just use other machines to connect to it remotely (e.g., http://[insert IP]:16000) and that has been my workaround while I figure out what is going on with Java. Java has been a world of hurt for FileMaker this past year and I'm hoping FileMaker is looking to get away from Java for the Admin console in future versions.
In addition to the info that you had from Taylor, you might overcome your Java issues by making sure that you clear the Java cache of legacy temp files. Dave Simerly kindly offered the following suggestion which helped me get the Admin Console open after installing FMS V220.127.116.115 on my MacBookPro running ML 10.8.3 . . . and then getting mad that the Admin Console was so damned fickle
Did you try opening System Preferences > Java, and then go into General > Temporary Internet Files > Settings and try deleting the temporary files? There may be some old versions left in there which are confusing things.
I hope this helps,