OK, now answered for any other poor saps who may want an answer to this too:
FMS 12 only works on Windows Server 2008 SP2.
Server was running SP1. SP2 is available as a download but for some reason wasn't automatically downloaded by system updater. I manually downloaded SP2, installed it and FMS 12 then installed fine.