Thank you for your post; apologies for the late reply.
I'm unable to replicate the issue with this information alone. If you can provide more detail, it would help. I have FileMaker Server 11.0.4 on OS X Server 10.6.8.
Here's what I have done:
1. Two Open Directory groups: Group1 and Group2
2. I have created two Externally Authenticating Accounts in FileMaker Pro for my test database: Group1 and Group2
3. The Accounts in FileMaker for the file are assigned privilege sets: Group1 = FullAccess, Group2=ReadOnly
4. Uploaded the test file to be hosted by Server
4. Test User member of Group1 in Open Directory group manager, opens hosted file from client and has FullAccess.
5. Delete Test User from Group1, put in Group2 in Open Directory, open hosted file from client and Test User has Read Only access.
I didn't restart the server, but was able to have the Test User's privilege set change by changing Open Directory groups. Please let me know what you have done differently so I can adjust my testing accordingly. Thank you.
I am having the exact symptoms as MV2011. I followed TSTuatara's workflow exactly to reproduce the problem generically outside my system.
FileMaker Server 11.0.4 is installed on 2x2.8 GHz Quad Core Intel Xeon xserve running Mac OS X Server 10.6.8.
Just following up: the FMS server was on a separate machine from the OD server. Having them on the same machine is probably the deciding factor there.
I need to spend more time checking it out, but FMS 12 may be the same situation.