Is there anything in the system logs that indicate the problem?
This is longstanding bad design, a bug, in implementation by the FileMaker app developers: a Mac OS X application should not behave like that and it's strictly against the Apple HIG. So far they have not fixed this bug, but hopefully it will be fixed in a future version, hopefully soon.
This improper and unprofessional app behavior has also been reported/discussed in the following threads (that I know of):
The post at the first link contains a quickie AppleScript that I wrote that you can use to workaround the launch-based problem on your XServe. It doesn't work around the other related issues (bouncing in the dock during scripts and progress dialogs), but those are generally non-blocking, just stupidly annoying.
and, here we are with FMP12 in 2015 and it still won't restart properly on a mac. whether I use login items for the database, or just select reopen apps on restart, fmp12 opens a file dialog in the right directory and goes no farther, it just waits for me to click 'open'. this ought to be easy to fix, it ought to embarrass Filemaker Inc. and Apple, but apparently FMP developers have no shame.
More like FMP13....another version and another ignored opportunity to fix this very wrong and longstanding misbehavior (and the other related misbehaviors of pausing in the background and bouncing in the Dock).
Even Microsoft's apps manage to get it right, but FileMaker ("An Apple Subsidiary") can't seem to, so these silly AppleScript workarounds continue to be necessary.
I can't believe it's been almost 4 years since my post, with zero action or acknowledgement of this bug. "Embarrasing" is indeed the correct word for it. The ghost of Steve Jobs needs to visit the development lead tonight and do "A Christmas Carol" until they change their ways and fix this silliness. ;-)
I must agree. This has annoyed me for years, and it has made it difficult to automate a number of business process. Why, why, why can't they fix this??