FileMaker 5.0 was introduced in 1999 - way too old for OSX 10.9. There is no support for PowerMac Software (what FM 5 is) anymore.
If You have FM-5 files: You have to migrate to FM11, then from FM11 to the current version of FIleMaker (FM13).
The last OSX to run old Mac-software was (IMHO) 10.6.x
FileMaker 12 and 13 are running fine under OSX 10.9.x
Thank for your help Markus !
If you're using any virtual machines (Parallels / VMWare), there are chances that FM 6 (maybe FM5) keep's running an at least Windows XP, maybe even on Win 8.