It should stay open unless some action by you or a script is closing the popover. What is causing it to close?
Clicking outside the popover will close it--which you want to do apparently. A script can cause the popover to close if it changes layouts to perform a task and then returns. If this second case is what is closing the popover, give the popover panel--not the button, an object name in the Inspector's name box and then add a go to object script step to the end of this script that causes the popover to close. This actually re-opens the popover, but in most cases in this type of script, you will never actually see it close and thus the popover appears to stay open.
I am using the popover to add a line item to my invoice table.
And how exactly does it do that?
This sounds like what I described earlier, a script that changes layouts, performs a task (creates the related line item record) and then returns to the original layout.
The layout change will cause the popover to close automatically, but by using go to object as the final line of the script, the script can return the focus to the popover panel and this will re-open the popover. To the user, it will be as though the popover never closed.
You may be interested in a popover example used to add line items found in "Adventures in FileMaking #2 - Enhanced Value Selection". One of the popover examples in this file allows you to specify the quantity directly in the popover and this user action creates the new line item record without using a script to do so.