Well you should be able to install server on the new box as long as you are not running it at the same time as your old box. If you have any files that have an explicit file reference to your old box you may want to keep in mind to update them.
Also any shortcuts you have in any opener files if you have them or Open Remote should be updated.
Overall I dont think it would much much of an issue. you can just copy over the files over to the new server and shut down the old server to do your testing.
One additional thing that I neglected to mention. When you copy your files make sure that they already either from the backup copies from FileMaker server or that you close the files first and stop them from being served before attempting to copy them.
Sometimes these things are just second nature to one that has experience and we forget that not everyone who reads these threads have this experience so we should mention it. ;)
Are there any rules or steps to follow that i can reffer to for the process.. Does it have any white paper for the pricedure?
I dont think there is a white paper ( well at least one that I can think of off the top of my head ) that covers this. There is one for Paid TechNet members that covers cloning as a backup but nothing about migrating to a new server per se.