I don't think that FMS12 is certified for Windows 8 (http://www.filemaker.com/products/filemaker-server/server-12-specifications.html) --for example, I know it doesn't work on Windows Server 2012.
That said, if the install works fine except for the SMPT test, have you tried playing with Windows Firewall? (I assume that the SMTP server on a separate machine?)
Firstly, just try turning Windows Firewall off, and deactivate any other antivirus/firewall software.
If you manage to get your required SMTP connection, you can then try switching parts of the firewalling back on (to maintain security). You should be able to restrict the open ports to the required ports for FIlemaker (http://help.filemaker.com/app/answers/detail/a_id/10672) and the basic SMPT ports (usually 25 and 587).
Thanks for your answer, FMS12 worked fine with Windows 8, but this was a fresh installation with Windows 8.1.
The installer wouldn´t start unless I choose "compatible mode", but then it was installed just fine.
Java is still a problem, so I need to use my mac to controll it. It could be something there which makes the SMTP test to fail.
Firewall didn´t help...
I will wait for an update which makes Java and Windows 8.1 OK... :-)