With the approach that you've tried, you'd change that dialog by not allowing users to override the validation error. In a script, you'd trap for the error and revert the record when the value is not unique.
But here's a simpler way. Define this relationship:
Table1::Species = Table2::Species
allow the "create" option for Table2 in this relationship.
Now this script step, executed from a Table1 based layout:
Set Field [Table2::Species ; Table1::Species ]
Will create a new record in Table2 only if a record for that species does not already exist.
Thank you!! Works great...