Thank you for your post.
In the current design, we are relying on Windows to manage the window sizes. When a new window is created, Windows links the original window to the new window, so when the new window is resized, it takes the original window out of maximize mode. This does not occur under Mac OS X.
Our Development and Testing departments are aware of this issue, and I have attached your post to the original report.
Taking the window out of maximize is expected. Getting a completely unexpected window size is not. It's that pop to whatever size the window had the last time it wasn't maximized that causes the most trouble.
Of course, eliminating the "window inside a window" look to unmaximized FileMaker windows on the Windows platforms would also be a nice improvement.