Make a calculation field that concatenates values that determine the unique quality of the data. Set the validation to be unique and capture error 504 in any scripts. if this isn't enough info I'm sure you can look up additional questions in Help.
Specifically, make that a text field with an auto-entered calculation such as:
Field1 & Field2 & field3.
Then switch to the validation tab and select the Unique values option for this field.
Thanks for the extra info Phil. I was trying to figure out how to set validation rules on a calculated field. A text field auto-entered by calculation, who'd have thought would also exist?