If you are going to upgrade, I recommend upgrading to the most recent version of Filemaker. You can give this a test by downloading the 30 day free trial version and trying to convert your files to Filemaker 10 format. This can be done, though there are likely to be issues you'll need to fix after the conversion.
Heres a knowledgebase article on the conversion process:
Switching over from FM5 to FM7 or higher is a big pain. The documentation is very big. If you have a single relational database as your structure, it shouldn't be as difficult. The new filemaker has a conversion tool but it's pretty pathetic. I tried it on my database and the errors were atronomical. I tried to rewrite my database but after a few weeks, I decided to stay with FM5. (I have 15 relational databases with several hundred scripts).
I guess it depends on your database files.
I've converted over 50 related files from FMP 5.5 to FMP 10 without too much trouble. There were a few fixes, (Window control scripting, adding account names, about 4 calculations that had to be fixed and one text based relationship that didn't work unti I cleaned up the data), that had to be made, but making the change wasn't all that bad.
On the other hand, merging those single table files down to less than a half dozen multi-table files... that was a major chore and not one I'd volunteer to do again any time soon!
When converting from FMP 3 thru 6 to FMP 7 thru 11, there are several mistakes easy to make.
The FMP ver 3 to ver 6 will be multiple databases that are related.
The conversion to FMP ver 7 to ver 11 will convert the separate databaase to updated separate databases.
First - not SAVE AS smaller while in the earlier versions.
This can clean up things and better prepare for the conversion.
You NEVER want to convert the working databases, ONLY copies, better Save As Smaller Copies.
Second - not SAVE AS Clone (without any records)
This can clean up things and work better than SAVE AS SMALLER for conversion purposes.
Third - not Recovering the existing database and trying conversion on the RECOVERED Copies.
This can work when the first two do not. (copies - not the originals)
[problems that accepted in earlier versions can cause major problems in later versions]
Selecting only one database of seven (for instance) may not work as well as
selecting ALL databases and selecting Open With FMP 11
This can cause all databases to be updated at the same time. (not always, depending on th OS)
Short answer is no, FileMaker 5 is not supported in Vista.
As Phil pointed out, since you'd need to go at least to 8.5 for vista, you might as well go to the current version 11.
Conversion is not that big a deal. There are a few common "gotchas" to watch for, but if your solution is not super complex you'll be up and running quickly.
There is a tool called MetaDataMagic that is essential if you really want to do it right. It cleans up your file references and guides you through any potential problems.