First off, there are a number of documented warnings about running FMS on OSX server. FMS documentation and help to start. Next, there are a number if workarounds. Yours for the searching. I believe the conflict is duplicate Apache implementations and port conflicts..
I am believing something is wrong with 7.51. I just posted a problem using 7.51 64bit on a windows 7 server. Warnings about a bad certificate and then no access to the Console at all. Even trying to go back to the previous release did not work. I had to do a complete reinstall of FM Server 13.
I found that installing 32-bit Java 7 update 51 on a Windows7 Enterprise SP1 PC also broke access to the Admin Console on FMS 11 server running Windows 2003 Server for me. I do not recall any error message. I believe that it simply stopped launching the Admin Console. I had to uninstall update 51 and revert to Java 7 update 25 to get access to Admin Console again.
You may need to add the IP of the computer hosting your server to the Site exception list in the Java Preferences pane/Security tab.
Click the "Modify the list of sites" button and add your FULL server address… something like :
… or whatever.
I just wanted to say thank you to Fred Bloggs. I had the same problem with Server V12 and by updating Java I got it fixed.