Have you installed FM on the windows-VM with the exact same license-key (LK) as you did on the mac itself? With a normal license you will get this error. Using the same LK and share data or the same server can only be done with VLA and FBA licenses, you'll probably need to buy another license
Sorry if I was not clear...when I said I said I had two separate "licenses" installed, that meant 2 separate "license keys", LK. That is not the problem. This looks to be a clear problem with Filemaker that the company needs to address. It looks to be a bug.
Thank you for your post.
Please take a look at Knowledge Base Article #6594
The key statement is:
"You may not run or use more than one instance of the Software on the same computer (e.g., using virtualization or other technologies, such as Parallels, VMware, etc.) at the same time."