Thanks for posting!
You are using the proper format in domain\group name; I think the issue may lie elsewhere. In Manage > Accounts and Privileges (assuming you have Pro 9 here), change the 'View by' drop down menu to Authentication Order. Move the external authenticated accounts to the top of the list. Also, set a password for any existing FileMaker authenticated accounts that do not have a password (example: set a password for the default Admin/no password account).
In FileMaker Server, what option do you have selected underneath Admin Console > Database > Security > File Display Filter?
Let me know how it works out
It appears you are correct about "somewhere else". Please note that this issue is Login to FileMaker Server Admin Console. I have looked at the Windows Security Log - it shows the following: 1) If put unqualified ad userid and password. Trys active directory authentication first, if successful in security log, the login will show fail from FileMaker Server Admin Console. 2) If try local account (not qualified) an attempt is made to Domain first, then local, succeeds and completes login. 3) Creating a local account with same userid and password as domian and try login unqualified authenticates to AD, but fails Admin Console. 4) Try login to userid qualified to local (servername\userid) authenticates to local and succeeds.
We do not have FM9 Pro Client, can't find Authentication Order in FM8.5 Pro. File Display order is "List All Databases"
My theory is that Server Admin Login Processing has a problem with AD account after authenticate. Neither the domain or local accounts tried are defined as filemaker database users.