What operating system is your computer? There are system settings specific to your computer's operating system that control what application is used to open a file when you double click it.
In the mean time, see if you can launch the FileMaker application and then use Open from the file menu to open your files. I predict that approach will still work for you here.
I'm on system 10.6.7 and using Filemaker 10.0.3.
If I use Open from the File menu it thinks the files are from an older version of Filemaker. They aren't. It's the exact same version. It then wants to convert the files. Yes I suppose I could do that but since it's not even recognizing it's own files I would kind of like to address that rather than just bandaid it.
I'm not a Mac user, but if memory serves correct, you can select one of these files and use Get Info to look at info about that file. I think there's a control in the Get Info dialog that specifies what application will launch when you try to open these files and a way to apply this change to all the files. I'd fix that issue first and see if that corrects both problems or if you still get a request to convert the files.
I did that and it think the files are old ones and wants to convert them. Doesn't fix it just changes it from one problem to another.