I'd never heard of Nutshell until you mentioned it, so I did a bit of Googling and you're quite right, Filemaker did indeed begin life as Nutshell. But over the last 20+ years, Filemaker has developed into a product which, from what I gather, is far removed from Nutshell. You won't be able to simply "upgrade" from Nutshell to Filemaker. You'll need to export your Nutshell data into a format that can be imported into Filemaker.
Filemaker comes with several pre-built "Starter Solutions", such as contact management and basic invoicing, etc, and these may be perfectly sufficient for you, but the real strength of Filemaker is that it allows people to build their databases from scratch to suit their specific needs. If you're not sophisiticated in computing then you may need to find someone who can give you some hands-on help with migrating your data from Nutshell to Filemaker.
Filemaker is a terrific piece of software, and I highly recommend it, but it may not be the best solution for your specific situation. I would suggest that you download the trial version of Filemaker, test it out and get a feel for it. But no matter what solution you end up using, the first thing you'll need to do is get your data exported out of Nutshell.
Backup your database and then back it up again.
In the export options in Nutshell or Nutshell Plus you may have these options
allows for the following formats, ASCII, dBASE II, DIF, Merge, dBASE III/IV, or Filemaker.
Exporting the data in each of the available formats would be safest.
You can run a Virtual Machine in Win7 - Virtual PC, Parallels, VMWare, VirtualBox and install WinXP to continue to run NutShell.
The best option would be to move the data into a modern database.