I really have no idea... but just having battled an install of FMSA12 on a new Mac, I suspect there might be an issue with your Java. Check out the Knowledge Base at FMI's web site for v12 as there are quite a few mentions of similar windows issues.
I suspect hardware/OS issues.
Since you're using 2003 instead of 2008, have you verified that your system meets the requirements at:
server 2003 64bit isn't even on the list of supported OS (only 32 bit server 2003 is on the list)
If you're not running a dual core processor (IE single core XEON), that could be causing the issue. the fact that the error message points to a configuration problem tends to push that.
Not sure if you're able to go back to 32 bit 2003, or forward to 2008 64 bit, but I would seriously consider using a modern OS. Even testing it on a windows 7 64bit box to see if you can get that working.