We are currently running version 126.96.36.1990 of FMS11 on OSX 10.8.5. This was a pain to install as we had a hard time getting the Admin Console to open up in the first place. We eventually discovered that we needed the latest versions of Java SE 6 as well as 7. Anyway, we eventually got the Admin Console to start up so we could finish the install and everything was working perfectly. After closing the Admin Console, we were unable to get it working properly again. Normally we would run Admin_Console_Webstart.jnlp which the Start Page had us download. Now whenever we run Admin_Console_Webstart.jnlp we receive the attached error. The server still seems to be running fine, however.
Using the 'latest' Java SE is probably the culprit, there have been a ton of issues with java versions and FMS11/12 admin consoles. I would uninstall those latest JREs and use the one that comes with FMS11.