Did you try to convert directly with FMP 11 ?
Two weeks ago i converted an fp5 file with FMPA 11.0v4 successfully.
Let me know if i missed something.
FileMaker 6 and older won't run under current versions of OSX
If You have .fp5 files, You need FMP11 to convert them to .fp7, then open those files with FileMaker 13.
Thanks Marcus, I tried what you suggested and it worked fine.