It should definitely work! Users using Terminal Services rely on it. I've not seen that error. I would actually try contacting FileMaker support, they might be able to help.
Filemaker server is not required to host a database. Filemaker Pro can host a database to be shared and is the only peer to peer networking you should use to share a fmp database. Don't use WIndows Server sharing to share the database, it will cause corruption.
Open the database in FMP, turn on sharing, setup security, then use Open Remote from the other FMP client.
Yes, the only problem is... My VPS is a windows server machine. And Filemaker Pro will not install it self on it. it keeps looping the installation file. So yea, i can' get futher then that. ;<