Did you try this expression?
( Pertussis Infection maternal ≠ "Yes" and IsEmpty ( blood test swab ) ) or
( not IsEmpty ( blood test swab) AND Pertussis Infection maternal = "Yes")
If the custom validation error message wont' work for you, (you can only use static text and it displays for all validation errors), you may need to use a script trigger to perform a script so that you can use the much more flexible show custom dialog to provide the user with a meaningful error message. (There's also a revert records script step you can use in this script to roll back the edit.)