This has been a "thorn in my side" issue for a couple of Filemaker versions. So I took the time to document it and send it to filemaker. So that others can see this issue I figured it would be good to share the bug report.
Product and version: Filemaker Pro Advanced 14.0.4 (and FMP14, FMP13, FMPA13...)
OS and version 10.11.3 (10.11.4, 10.11.0, 10.10.5...)
Description: When zooming a window (option pressing the green zoom button) on a window that is not the main one has very unpredictable results. Same goes with using the Script step Adjust window Resize to Fit.
How to replicate: Attach 1 or two extra displays to your computer. In my case I have a total of 3 displays. My Laptop retina display set to 1920x1200, A Large display arranged right above the main screen at 2560x1440 and a 3rd screen arranged to the left of the Retina is 1440 x 900.
Open a filemaker file and have it display on the Laptop retina Display. Option zoom (Standard green zoom button)it and it will correctly fill the screen. (Don't just zoom it as we are not talking about Full screen issues here). Option zoom again and it will go back to it's original size.
Drag the window to the top screen (2560x1440). Do an option zoom again. It will zoom to the correct size BUT the top is not set all the way to the top of the monitor. If you drag the window to the top of the monitor you can see it is the correct size, but it didn't set it to the correct position. (I'm guessing it is using the clipping based on the height of the main display 1200) . NOTE: Youi can not Option zoom again to get back to smaller size again on the same window as doing that makes it maximize to the full size ON the Laptop. So manually size it smaller.
Drag the window to the left (smallest) display. Option Zoom on it. OOPS... Now it once again sizes the window correctly, but now the window is partially off screen as it has set the top location of the window totally wrong. If you drag it from the side you can move it back onscreen and see it is sized correctly, but it's position was wrong.
NOTE: Similar things happen with just two monitors also. Moving the monitors to the right gives "different" but wrong results also. On some OS's I think having monitor on Right actually worked, but not any recent OS.
Do the same thing with Apple's Mail.app and it properly zooms. So this IS a filemaker bug. (Which is why the Adjust window Resize to Fit also fails)
Attached is the screen arrangement.
Workaround: None (Me: Dr it hurts when I do this....Dr: Don't do that)