An example of the syntax is:
open file "Hard Disk:Databases:Addresses" with passwords "your_password" for accounts "your_account"
To whom it may concern.
It may be late, and you have figured out what your problem was, but I got the same error and in my case it was that the path to the file had spaces in one of the folders and I had not converted them in acceptable Applescript format. In my case, I had .../files to convert/... which didn't work, but when I did an easy fix like .../files_to_convert/... it worked fine.
This is kind of a different method, which is useful if the file is local to your machine. Which is to create an AppleScript file which you just drop the file on. It does not need to know the name of the file. If I need a different password/account for a different file, I just duplicate the AS file, make the changes, and put it in the folder with that other file. It saves having to add these to the Keychain Access app's list(s):
on open (theFiles)
tell application "FileMaker Pro Advanced"
open theFiles with passwords "pw" for Accounts "admin"