When a button bar segment is defined to be a popover button, and the popover is selected, it is nearly impossible to cause the Format Popover dialog to appear or to get its corresponding menu item to be visible. Instead, what constantly appears is the dialog for configuring the button bar, with segment 1 being the one that is considered current, even if the popover button is assigned to a different segment.
I have occasionally been able to trick FileMaker 17 into showing me the expected dialog, but I can't recall how I was able to do it. Surely this kind of accidental luck can't be what users must rely on.
In addition, another confusing point is that both setup dialogs, the one for the button bar AND the one for the popover, are both titled "Button Bar Setup."
One more thing: it works fine in FileMaker 16.