I am answering my own question here, according to support, apparently the latest version of Filemaker Server 11 is incompatible with OS X 10.6.3, it's the daft Java console again. Why of why do they even bother with this Java crap. Very, very pissed off. Come on, Apple owns this company.
Hmm, that sounds odd... FMS11 can be installed on a Mac OS iff the OS is 10.6, according to FileMaker Server's tech spec page