Have you try "Recover" under file menu?
OSX? a Google for the error message brought up this
A FileMaker Pro application has Instant Web Publishing enabled. FileMaker Pro is then closed. To launch or open the application, you drag a database onto the application icon or the user double clicks a database.
In previous versions, launching FileMaker Pro in this fashion would produce a port conflict dialog.
In FileMaker Pro 11 and later, the port conflict dialog appears along with an additional message:
“The document “<file name>” could not be opened. FileMaker Pro cannot open files in the “FileMaker Document” format.”
This is an issue that is currently be investigated and does not have a set resolution at this time.
However, please note that the issue only occurs when FileMaker is closed and the drag/drop or double clicking a database methods are used to launch it. As a workaround, launch FileMaker first through the Applications folder or the application icon within your dock and then drag/drop or double click a database to launch a file.