Use FileMaker Pro Advanced with the Debugger turned on.
there is IMHO no way with FileMaker Pro/Pro Advanced, although there are some installations that din't show that dialog (older versions of FM,), but then, something turned wrong
if You need a FM to start 'quiet', use a 'starter-file' with no functionality, just as a dummy. When opening that file by double-click or via start-items from the OS, it should work..
I misunderstood your questions, JohnLeonard. Now I see what you want. Actually, I create short cuts / aliases via URLs for FileMaker and that opens up solutions directly into FileMaker and I believe it will skip open file dialog for back to version 11. It is hard to remember that far back. But it clearly works well in FileMaker 14 using the new FMP URL.
Any chance you are considering upgrading? FM 11 isn't supported anymore which means not only no bug fixes, but also no security updates.