There are two levels to this issue:
1. On the data level, you must have field validation to prevent duplicate combinations. It looks like you have this solved(although a simpler calc would be Product ID & "|" & VendorID).
2. On the presentation level, you should have a selection mechanism that will offer only remaining combinations for users' selection and prevent validation from kicking in. Note that this is not "another way" but an addition.
There are several ways to build such a mechanism - it really depends on the selection method (value list, portal, list).
So the only way to circumvent the double validation dialogs that FileMaker presents is to use a Presentation approach that disallows the validation from running at all?
I guess this might like a filtered value list that shows only products that haven't yet been released for a given vendor, where I would have to explicitly change a value in a field to filter out a given Product (because the release form was signed for). Then also, this allows for reverting a form when the form is flawed or lost, et (by changing the value back).
I like this idea better than just using a button that takes over the whole process. The button only approach seems too vague and implicit and doesn't give a sense of control.
Yes, you want to prevent any kind of validation dialog, double or single, because it's confusing to users. I'm afraid I don't quite follow the rest of your post. If you want to use a value list, you indeed need to filter it to include only unreleased products. This can be done by defining a relationship based on a calculation field listing all released products' ID's, and using the ≠ relational operator. There's a bit more to the technique, depending on your version - see: