Well that's how we did it all the time before we had the option of the new window command.
Go to Layout has a "name by calculation" and a "number by calculation" option. Thus, you can use Get ( LayoutName ) or Get ( LayoutNumber ) to store the current location in a global field or variable before you change layouts. Then your "back" button uses one of these "calculated" go to layout options to return. You can even use a return separated list to set up a "stack" data structure in that field or variable so that multiple clicks of a "back" button retraces your navigation through several layouts.
Ps. While less than thrilled with how FileMaker manages multiple windows on a Windows OS--neither maximized windows nor unmaximized within an "application window" makes for a smooth interface, I've been able to minimize the size changes with careful scripting. You can find some examples of my efforts in this regard in the Known Bugs List database.
I've copied parts of your script and have popups working fairly well. I have a weird issue on Windows platforms however. When the popup window is closed, as the main window is Maximizing, my startup layout flashes (a list view layout from a different table). Have you seen this before?
I have not. And I would call "working fairly well" a very accurate description of this method as it doesn't elminate all the issues, just minimizes them.
Is that startup layout open in a different window or does the main window layout "flash" this layout?
If the Main window is "flashing" this layout, you may be tripping a script trigger that performs a script that changes layouts like you are observing. There are ways to keep that from happening if that's the source of this trouble.
Main window "flashed" this layout. I'll check my script triggers