1.Did you set your server at the first step of install Windows OS to be belong to the Domain Server?
2.Did you set Security as 'FileMaker and external server accounts' which appears in Databases>Security?
Directory service settings is not for authentication, its purpose is that
'When FileMaker Server is registered with a directory service, host names and IP addresses are published to the directory service so that FileMaker Pro clients can use the directory service to find the server on a network.'-refer from FileMaker Server Help.
I understand FM server may form a search query that possibly looks like this
&(objectClass="People";AccntName="ADAccount") and sent off to Directory Services. I am suspecting for role based accounts, the objectClass may not be people (its a Role) and due to this the authentication fails.
The question here is
1. Does FM server forms a query string that is sent to Domain Controller? If not how does it authenticate existing users of a domain.
That theory is completely faulty.
(If this indicates that you also use the 'directory service' tab on FMS: don't do that. That tab has nothing whatsoever to do with authentication the user and can only lead to confusion like this).
The exact mechanism of the communication between FMS and AD varies depending on the client's setup. If the client is already logged into their workstation using AD credentials; whether that workstation is Windows or not,...
But regardless of all that, the actual communication to the AD is not done by FMS but by the operating system of the FMS machine so it follows the protocol of the OS. There is nothing to intercept or change on the FMS side.
Why have local groups on the FMS machine in this AD scenario? Why not create groups in the AD? This mix and match can not make the security folks happy and would make the deployment fail a security audit in most organizations?
Can you provide a little more detail when it comes to group names and group memberships?
How is the authentication order in FM set up?
Any chance that there is a matching group by that name in the actual AD? If so that would make it fail if the user does not belong to the AD group by that name.