Did you clear the "validate ony if field has been modified" check box?
Yes - no difference.
Try putting the name of the field in the validation.
I tried it with validate by calculation and just entered the number 10 in the fieldname (auto-enter by calculation and replace existing number unchecked) calculation box and and got your results, but when I put the name of the fieldname=10 in the box it rejected a non 10 result. Didn't have to time to research any further but HTH
For current versions of filemaker, either
Self = 10
Total = 10 /* assuming field name of this field is "total" */
For this type of auto-entered calculation, the check box "Do not replace existing value..." should be cleared in most cases.
Here's a working demo file: http://www.4shared.com/file/259089560/3a82a207/AutoCalcValidation.html