Have you checked the security log on the W2K3 server running FMSA11? It may have useful information about why the logon failed.
Would you be willing to try a small experiment? Try setting up an Administrator Group via the admin console that uses an external group in your OD and see if you can log on to the administrator console using an account in that group. Most likely you will have to type the username as email@example.com or odshortname\username
If that works, then I suspect you have the same problem as us. It appears that FMSA11 uses interactive logons for administrative access. Interactive logons will obtain kerberos credentials, which I think is what OD uses and are also used in our hybrid K5/AD environment, thus we can use administrator groups successfully. However, the client connections appear to use network logons which try to do a simple logon that doesn't fetch full credentials and causes the logons to fail when they would require kerberos credentials, thus our client logons fail.
I attempted to perform the test you described but I cannot create an admin group that uses an external group because I cannot bind the Win2K3 server to Open Directory without demoting it from being a Domain controller and I don't want to do that.
My scenario is a bit different than the typical in that I'm trying to get away from Macs authenticating against Active Directory. I work at a school and I'm pushing things towards nearly all Mac with a few exceptions. I have a Win2K3 server that I'd like to use to host Filemaker since it's a sunk cost and my department's budget is regularly being reduced. In spite of that, I'd like for the Macs which will be authenticating against Open Directory to be able to access Filemaker without having to log in.
I'm guessing it would be necessary to bind the Win2K3 server to Open Directory (functioning as a Windows domain controller) but something tells me that won't be possible without demoting the Win2K3 server from a Domain Controller. Or, perhaps the easier answer is to find a Mac to run FMPSA on.
I didn't notice if you set up your Directory Service Settings in the Filemaker 11 Server Administrator.
Database Server>Filemaker Pro Clients>Configure Directory Service Button.
Try these settings:
Replace <domain> with yours.
I don't know how your network is set up, but it works for me. Just be sure the computer logging on to the FM database is logging into the domain on startup. Then you wont be prompted for a password in the filemaker database. If someone logs in with external authentication, then Get(AccountName) gives you their real account name in the external directory server. The Get (PrivilegeSetName) function returns the name of the privilege set assigned to the current user. If you change the name of a privilege set then you have to modify scripts too.