You mention two different versions of FileMaker in your post. And FileMaker 12 cannot host files created by FileMaker 10 until they are converted to FileMaker 12 format. Have they been converted? Can you access the sample file that comes with server?
Do you get any error messages?
Can you get the server admin console to open?
What do you see when you open the console and click Databases?
The issue with Server 10 and SBS is that sharepoint--an SBS feature can conflict with Server. SBS works with server 10 (Or so I understand) if you can remove (Disable?) sharepoint on those systems.
We have server 10 successfully operating on Winserver 2008--a 64 bit machine so I would expect server advanced 12 to also function successfully on a 64 bit system.