Don't do it! One of the servers will shutdown when they detect each other on the same network and it's random as to which server will shut down.
Join up to the "Filemaker Developer Subscription" found on the rhs of the home page here and you will get a 3 user license of FileMaker server that you can use for testing.
If you can post some specs about both machines then I can give you an idea about basic performance, but new vs old, the Win machine is likely to perform better because of age.
What if i move one machine on a different network?
old mac is running on 2 intel Xeon E5462 4-core cpu @ 2.8GHz
the new win machine is running on a single Xeon E5 6-core @ 3,5GHz
It won't be detected on a different network so that is okay... It depends on how critical the files are!!! It's very easy in this case to make a mistake, e.g. forget to prevent FMServer service auto-restart and move the server onto the network... etc. so many things can go wrong at the wrong time.
The FDS is about $100USD IIRC and it's useful going forward if you want a development server.
Your old mac looks pretty good, I can't really even make much of a general comment on the gain you would get.