I can see why you've been struggling. This is harder than it looks.
In Activity, via calculation field:
Gather IDs of all people who've done THIS Activity, via the normal relationship to the join table, using List().
Use the above in a ≠ (does not equal) relationship to a table occurrence (TO) of People.
Result will be all people who've not done THIS activity.
P.S. If you downloaded the file a few minutes ago, download it again. I made it much simpler.
Thanks, I had no idea that List could be used in this way. Its made this and a lot of other things possible.