A screen grab may help.
Also, do you know what video card is installed on the server ?
I have come across something which sounds similar on my own system, it turned out to be a video card driver update I did.
Thanks for the reply will look into the video card and what driver it currently has
I'm not sure I understand the issue. You mention that the FileMaker Pro screen goes crazy, but the specs you give are for the Server machine, not for the Pro client.
If you are indeed talking about Server, then XP just won't do it. http://www.filemaker.com/products/filemaker-server/server-10-specifications.html