FileMaker Server is currently incompatible with Java 6 Update 10 (or later) on Windows. So you may need to unistall your version, go to the Java website and pick up a vintage version.
Also opening up the Admin console on the Server is a known problem with FileMaker. Its ok when I access it on a client pc's. (Done with your server address and port number 16000 opened up.) However - for me this even times out after 3-4 minutes. I have logged this with FileMakers' Tech Support too.
The FMS11 installer I have comes bundled with Java 6 update 16