If your filemaker server 32 or 64 bit? I know there's two different versions of the filemaker drivers based on what version of FMS you installed. And you also need to use the 32-bit Data Sources manager if you have 32-bit FMS on a 64-bit OS.
Is mathematica/DSNs on the same machine that is hosting the FM file?
Here's all you need to know about DSN differences:
I'm currently connecting Mathematica 10 to FileMaker Pro 14, Java 8 (1.8.0_45), on a Mac Yosemite 10.10.3, but I've had success in the past with earlier versions of Mathematica, FileMaker, OS X, and Java. The biggest issues were always getting the FileMaker JDBC driver into Mathematica's Java class path, and then getting the connect string correct.
Here's how I do it:
I put the FileMaker JDBC driver at the most global level:
I use the the most verbose version of OpenSQLConnection
In:= conn = OpenSQLConnection[
"Name" -> "FMP", "Username" -> "Admin", "Password" -> "password"
Out= SQLConnection[Name: FMP ID: 1
Status: Open Catalog:
Oh, and in the ODBC/JDBC Settings, I have Sharing turned on, and I specify users by privilege set -- for my purposes currently security isn't an issue, I'm connecting using Admin which has a Privilege Set of 'Full Access'.