Welcome to the forum.
I've been able to successfully convert over 50 FMP 5.5 files to fmp 10.
I'd try using your FMP 5 application to recover the file first. Sometimes there is hidden corruption in the file that doesn't come out of hiding until you attempt to convert the file.
how do I find or do the recovery application for FMP 5?
Launch the Filemaker Pro application but do not open the file you are going to recover.
Select Recover from the file menu.
Select the file you want to recover.
The recover process generates a new copy of the file with "recovered" as part of its file name. You can then try converting this copy to see if it will convert.