In layout mode the show/hide screen dimensions menu contains a "custom size" menu item. Selecting this item displays a dialog that allows us to enter a custom size. When a custom size has already been entered, selecting the "custom size" menu item displays a dialog that contains the existing custom size and when changes are made, the existing custom size is overwritten and destroyed.
I would expect to be able to add many different custom sizes. All mobile devices will operate in both portrait and landscape mode, so I see this as a bug or a very strong suggestion that FileMaker Pro development is not for mobile.