This section of the FilMaker Forum is intended for reporting possible software bugs. For assitance with using the software, please use the FM Pro Forum tab and for feedback "the edit relationship window is confusing", there is a feedback tab. I'll be glad to help you here, but posts to the FM Pro Forum section will be seen by more people and thus get more and faster responses.
Can you describe your files and their relationships in more detail? While Filemaker 7 - 12 supports multiple tables in one file, FileMaker 3, the first relational version was limited to one table in each file so I must assume that you have multiple files.
When you converted these files, did you convert them all in one group? (Launch FileMaker 11, select open from the FileMenu and then select all the files by control clicking so that all are opened/converted at the same time.)
In the relationship where there appears to be missing related records, what kind of match fields did you select for this relationship? Might you have text fields used to match values in the relationship? Between 3.0 and now text fields index values differently. In 3.0, punctuation and spaces were ignored. In FileMaker 11, they are part of the value being matched so small discrepancies in the text values can keep records from matching like they did in 3.0.
John D Smith will match to John D. Smith in FileMaker 3, 4 and 5, but not in fileMaker 11.
That's just one possibility, there are other issues that could be involved--even latent file damage that did not affect file behavior until you converted the file.
Bottom Line: Between 3.0 and 11 there were a LOT of changes made to FileMaker. The converted files are very unlikely to work perfectly until you track down and fix a number of minor issues.