You can't have one window Restored Down and another window Maximised – in Windows at least.
It's a pain, because it doesn't look very nice when a pop up window appears, your main layout window has to shrink.
I have used the following script step that makes the main window as big as possible to fit inside the instance of FMP without maximising, before showing the pop up:Move/Resize Window [ Current Window ;Height: Get ( WindowDesktopHeight ) - 4Width: Get ( WindowDesktopWidth ) - 4Top: 0Left: 0 ]
Then show the pop-up, then, when you're done with the popup maximise the window again.
It's not great, but, that's what I do when I need to.