You have two options that I can think of
1) FileMaker systems need not have all tables in one file to function. Your value added module can be an additional file or set of files with the needed layouts, tables, scripts etc. Those that pay for the value added version get this added set of files and those that don't pay don't get it.
2) Make an all in one system, but use data entered in a table not accessible to the average user to control what features are accessible to the user. Bascially, you have all the value added features in every copy distributed, but they're "turned off". This makes it easy to sell upgrades to your customers by issuing them a code that they can enter in a field to "unlock" the additional features.
While I don't have access to the proprietary info to know for sure, I suspect that the Runtime Version of FileMaker Pro, FileMaker Pro and FileMaker Advanced are examples of option 2.
One drawback to Option 1 I can think of is that, assuming the features in the additional modules will interact very closely with the existing 'base' modules, then the base modules probably will, equally, interact with the additional modules. So when you build the system and ship it with a missing additional module the user will get 'data source not found' error messages. Unless you can find a (practical) way around that.
Good point, but one that can be "designed around" in many cases, but it all depends on the details of the database design and any such modules.
I tend to prefer an all in one approach, even if you choose to use multiple files as it provides all users with the same exact structure which makes for easier customer support and maintenance.
And if Phil's speculation about FM installations is correct, one must reasonably give that credibility and think, "Well, if Filemaker does that, it must have something big going for it."