Runtime don't have license from FMI, it is from runtime developer. We can do nothing without knowing how the developer implement the license... is there plug-in (files with .fmx extension) in the solution?
As user19752 said, runtimes didn't have licenses like FileMaker Pro, but you needed FileMaker Pro to create a runtime. You can usually open a runtime with FileMaker Pro, but if it is checking for some license, you would have to interrupt the opening script to intercept its validation. That means you need the debugger feature which didn't come out until version 8. So maybe with version 8 you could convert FM5 to .fp7 and use the debugger to open it and stop the opening script from running. Just a shot in the dark. A FM consultant that does recovery could take care of this for you, but it will include an appropriate consulting fee. And once recovered, if the data is important, you probably will want a developer to bring the database to the current FM version. So expect to spend money on this assuming the data is worth recovering.