Your calc may need some tweaking:
Count (Prospects by Name::LAST NAME) > 1 and (DupeOkay = ""); "Duplicate LastName";
Count (Prospects by emails::EMAIL) > 1 and DupeOkay = "" ; "Duplicate Email" ;"" )
Remove the red bits and add the blue bit and see if it works.
But instead of using self-join tables and a calc field, an alternative technique would be to simply create a script that does a Find for duplicate values (using the "find duplicate values" operator, which is an exclamation mark) and then Omit records in which DupeOkay is checked.
Find duplicate values: http://www.filemaker.com/help/html/find_sort.5.9.html