in the Excel VBA routine, do you get an error ? Did you check in FMS log if someting appears ? If so you get a clue at why the file did not open.
And did yoy check the command line arguments filemaker.exe accepts ? You may for example invoke FilleMaker in your script with the database as an argument if possible. I also read, but never tried, that there is an FM object you could link to, an OCX object.
The command line arguments won't work but FM does have an ActiveX interface that you can call to open files. Do a search for ActiveX in the FM help for examples. Those examples are actually written in VBA.
In FMP16 Help, ActiveX refers to the knowledge base. From there I found an example of VBA coding in Japanese or something that looks like Japanese to me.
I searched on filemaker.com and found examples for FMP12 and 13. For 15, again it refers to the knowledge base. For 12, it states to add a reference to the FileMaker Library without naming it. Oh, the example seems to refer to VB and not VBA. I may have missed something.