Re-reading my post, I think the lightbulb turned on.
To see all notes, from all invoices, for a given contact, I'd need a new Contact TO directly related to the Invoice TO, using Contact::ContactID as the match field and Invoice::ContactIDF.
I tried it out and looks like it works: On my contacts layout, I see multiple Notes from multiple Invoices (regardless as to whether this contact was a Buyer/Seller/Broker on that particular invoice - exactly my goal).
So it seems I've answered my own question. I wrote this post for other peoples' benefit (and also in case there is something wrong with this answer which others can point out).