Perhaps you could provide some additional detail. Have you been able to complete a single machine deployment without web publishing enabled? Is the machine dedicated to FileMaker Server, or is it providing other services as well?
In addition, verify that there are not multiple versions of Java installed. If there is a 32-bit version and 64-bit version of Java installed, please make sure that the 32-bit version in listed first by dragging the 32-bit version to the top of the list.
When I search the internet for information on the java connection error, I find a reference to "no process is listening on the remote address/port".
The ports that need to be open for the server deployment can vary depending on the configuration.
Ports 80, 5003, 16000, 16001, 16016, and 16018 must be open. Ports 5013, 16004, 16006, 16008, 16010, 16012, 16014, and 50003 must be available.
Any other information you can provide may be helpful in narrowing down possible causes.