Using another calculation to calculate the age, which must be based upon the DOB field itself, and then asking for validation from that calculated age would produce a circular reference. Try validation as:
Let ( [
now = Get ( CurrentDate ) ;
then = DOB ;
thisYear = Date ( Month ( then ) ; Day ( then ) ; Year ( now ) ) ;
age = Year ( now ) - Year ( then ) - ( thisYear > now ) // must subtract 1 because not yet birthday this year
age ≥ 17 and age ≤ 64
Take your age calculation and use it as a validation rule on the birthdate field. An age calculation field must be unstored to update correctly as time passes so it can't be set up as an auto-enter field unless you only want to record the person's age at the time the record was created.
Your validation rule can be written like this:
Let ( [ Today = Get ( currentDate ) ;
age = Year ( Today ) - Year ( birthdateField ) - ( today > date ( month ( birthdatefield ) ; Day ( BirthdateField ) ; Year ( Today ) ) ) ] ;
age > 17 and age < 64
Note: If you copy this expression onto your clipboard, the inequality symbols will not paste correctly into FileMaker and you'll need to correct them before this expression works.