You're right, the "no portal inside a portal" rule cannot be circumvented.
You could use a generic popover for all portal rows, i.e. one where you display a portal into a relationship controlled by a global field - which you set with data from the respective portal row.
You could use a button to open a separate floating window instead of a popover. I would think the downside would be that if the user clicked outside the window, it would still be open but just be lost behind the main window. This might not be optimal. The trick might be to get the new window to close if a user clicks anywhere outside of it.
I can think of one technique for this. This is a method to create a script to close the window without the user having to click on a close button. It can trigger a script to close the floating window when the mouse moves outside the window, i.e. it will trigger a script on hover rather than click.
If you make a window to be a floating document window, it won't disappear behind other windows if the user clicks another window.
But I recommend the option described by erolst. To flesh out the details a bit more:
Place the popover button outside the portal. I usually make it very thin, use Hide Object When to make it invisible and stick it up close to an upper corner of the portal. This can create the illusion that the popover is opening up out of the portal. Give the Popover panel an object name. Place your second portal inside the popover panel and use a relationship that matches to the desired field in the second portal's table. Then put a button inside the portal row that:
a) set's a match field in the layout's table to a value from the first portal's table so that the relationship for the second portal matches to the correct set of records. and
b) uses go to object to open the popover panel.
I do like the erolst approach. So i create a global field and how it would open up portal based on that field.
Since you haven't described the tables and relationships involved, I will assume that you have the following relationships:
Portal1Table::__pkPortal1TableID = Portal2Table::_fkPortal1TableID
Define your global field in LayoutTable and create this relationship:
LayoutTable::gSelectedRecordID = Portal2Table|Selected::_fkPortal1TableID
Base the popover's portal on Portal2Table|Selected -- a second occurrence of Portal2Table
Your script would then have these steps:
Set FIeld [ LayoutTable::gSelectedRecordID ; Portal1Table::__pkPortal1TableID ]
Go To Object ["Popover Object Name Here" ]