As you know, this update is principally (until only?) a "java depedencies update"; so, one detail that may be very important is :
What is the exact Java version that is installed in your server ?
If you have installed the last yet : Java 7 Updade 17 at this time here :
, i can also recommend you to erase caches here :
System Preferences > Java > Temporary Internet Files > "Settings…" Button > "Delete" Button.
Then, try another time from the FMS Start page.
Another question :
Did you also try from a desktop computer ?
PPS : Excuse me for my bad english.
!!! SOME IMPORTANT UPDATE HERE !!!
I passed the 12.0.4 FileMaker Server Update and i have some updates to bring since my previous post.
The new FMS installer provide the latest version of Java (JDK version that seems to be a more complete version than JRE that i linked above, but no comment, i'm not a Java specialist).
So, normally, your server have now the brand new version of Java, unless you refuse its installation that was prompted at the end of FMS installation process.
BUT, i also noticed several problems with launching Admin Console on my Mac OS 10.8.3 Server and Clients. In this situation, the success keywords - for me - were : FORGET SAFARI, USE FIRFOX. I even recommand to define it as the default navigator on your server, before installing FMS 12.0.4.
Assuming that you had correctly installed FMS 12.0.4, following all detailed informations in the FMS Getting Started doc, the next step are :
First : If not done yet, erase Java cache files, as described in my previous post.
Second : If not done yet, install Firefox 20 here : http://www.mozilla.org/en-US/firefox/all/
Third : Access, with Firefox, to your FMS Startpage like http://[ServerIPAddress]:16000 and press the launch Admin Console button.
Fourth : When prompted, choose to "execute with Java Web Start (default)".
After few seconds, you will normally be able to access your Admin Console and you will found an alias on your desktop to relaunch this app when wanted. Because with Safari, i encounter many times a problem when the process was attempting to create this shortcut.
Good Luck !