What version of Filemaker are you using? In Filemaker 12, you have the option to specify the window with the "dialog window" window style.
In older versions, you can open the window, specify the layout etc. and then pause the script. With the script paused, the window will remain in front and the user will be unable to interact with the background.
If you are careful, you can do it like this:
Allow User Abort [off]
Then, any buttons or scripts that close the window, must include Halt Script to eliminate what is otherwise an infinite loop that cannot be aborted. (I test this without the allow user abort [off] step and add that step only when I have confirmed the other details of the new window script work correctly.)