I have a Mac Pro runninig Server (10.7.4) with FileMaker 12 Advanced Server. I have turned on Open Directory, making it a stand alone Open Directory Master, and set up the certificate and SSL. I've added Users and Groups and they all work fine for services on the Mac. I go to the FileMaker Admin Console, and Configure Directory Service. I put in the Server Address (ns2.mydomain.com) and Entry point (ou=ns2,dc=mydomain,dc=com) (obviously substituting my domain where I say "mydomain"), check the SSL box, and then press the Test Directory Service Button. It tells me the external authentication is successful. So it sure sounds like it is all working for FileMaker Server authenticating to Open Directory.
I go to to my FileMaker database. I create a group for external authentication. To keep is simple, I make the group "filemaker" and I give it Full Access and make sure Full Access has access by fmapp in the privilege set. The test user I created on the Mac Mini Server.app is "John Doe" and he belongs to group "filemaker". I try connecting and it fails. I've tried it with several names just to make sure and double checked passwords. I've tried it with the short name "johndoe". I've tried it with SSL turned off. I've tried going to Server.app and opening John Doe and making him able to administer the server. That didn't help. I've made sure that John Doe has the little globe next to his user name to assure he is an LDAP user. I've tried creating another group on another file on the same server, but it still doesn't help. In the FM Admin Console's Confi9gure Directory Service, I've tried changing the Address Point in the from the domain name to the public IP, then to the local IP, and then to 127.0.0.1. That didn't change anything. I tried it without an Entry Point, and that didn't change. Every time I made these changes, the "Test Directory Service Settings" button said the test was successful, but the FileMaker database still would not let the user in. I've made sure the "filemaker" group is at the top of the authentication list in the FileMaker Security so it is evaluated first.
In this past I have done this on Snow Leopard machines and it just always worked. This is just my development machine, but next week I'm supposed to try to set it up on a production machine and want to figure out why it is not working here first. Anyone else have a Mac Server 10.7.4 with FileMaker Server successfully using Open Directory authentication?