Sometimes one version of Filemaker will "tolerate" hidden damage to a file that only becomes apparent when you upgrade to a newer version. Try using FileMaker 4 to recover the file, then see if you can convert this recovered version. If that works, see if you can find an older copy of the file that does not require the recover to convert. If not, you may need to take your chances with this recovered, then converted file.
You also might try using version 4 to save a clone of the file to see if the clone will convert to version 11 format. If so, you can then export the data from version 4 as a tab or csv file, then import it into the converted clone.
If all elses fails, you can try exporting the data from your original file as tab or csv files and then import the data into a brand new FileMaker 11 file. This does not import any of your layouts or scripts, but at least you have imported your data.