Sorry, but that is not an option. Welcome to FileMaker for Windows as this is not a Mac issue.
Any time you open a new window all other windows will leave maximized state and resize to whatever dimensions were last specified for the window when it was not maximized. This is a major pain in the neck.
What you can do with a script is to open the new window and then resize the backgound window back out to "nearly maximized" dimensions. There are get functions that will return the width and height of your screen that you can use to resize the background window. Then, if you use a script to close the smaller window, the same script can revert your windows to maximized state. This produces a small but noticeable "twitch" in your background window. Some developers prefer to never have windows maximized as a way to avoid this, but personally, I despise the "window inside a window" look and the way two sets of scroll bars cause confusion for users so I choose the twitch. With Filemaker Advanced, BTW, you can give the smaller window's layout a custom menu such that closing the window by menu choice, keyboard short cut or the control in the corner of the window all runs a script that you can then set up to also maximize windows.
Fortunately, FileMaker 13 offers a new feature called a Popover. I much prefer to use popovers in place of most of the "dialog box" type windows I'd previously open whenever Show Custom Dialog was not a practical option. Since a popover is part of the current window, opening one has no effect on whether windows are maximized or not.
Thanks for you help, for the millonth time :-)
I'll check on the PopOver Option, first time I heard of it so i'll try to teach me how it works
Popovers are, in my opinion, the single most significant new feature introduced with the release of FileMaker 13.
I have a demo file that explores some of the creative ways that you can use one to enhance your user interface design: