Product and version
FileMaker Pro 13->16.0.2 (probably since 7.0)
OS and version
OS X 10.11.6
iMac (Retina 5K, 27 inch, late 2014)
Managing Custom Access to Layouts can fail to apply with a short error message "Used by Privilege Set" (translated from french) depending on what was the selection before Admin choose "Custom Privileges" on local menu. In English, after a try, sounds like the error message is more detailed but still make no sense to me, stating : "The current combination of privilege is not valid. It's not possible to both allow the creation of new object and disallow the modification of new objects".
How to replicate
Go to Managing Security
Create a new Privilege Set
Push all access "Up"; in particular, make sure the choice for "Layouts" is defined as "All modifiables".
Then Try to Customize access to Layouts, defining the last entry [All brend new layouts] to "Read only".
Workaround (if any)
Go back to the manage panel
Choose "All view only"
Then Go to Customize access to Layouts, defining the last entry [All brend new layouts] to "Read only".
There is a chance that testing / dev dept will say that it is expected behavior due to some circonvoluted reasons. If it is the case, just please responds them that as a reasonably experienced developer (10 years and 5 certifications), i took near twenty minutes to understand why i get this error. So if not a bug, it is for sure a bad design, but no, i will not create a new record in product idea area in order to expect a better design on this .