This is a common mistake made with aggregate functions such as sum and average. These functions cannot be set up to be used selectively. They sum or average all related values as controlled by the defined relationship.
As you have written it, the IF function controls whether or not the average function evaluates, not which values in the related table will be averaged.
If the first related record in Assignment Registration::category = "Understand", your average function will average the Mastery Level field for all related recordsin Assignment Registration, not just those in the "understand" category. If the first related record isn't in the understand category, you don't get any value returned.
A summary report based on the Assignment Registration table can be set up where you group records by student, then by category by sorting the records, then a summary field can be used to compute the average for each category.
To add, I have tried to create another calculated field using the "List" function. It did succeed in listing only the values that I needed to average, yet I am unsure how to average all of the values in a return-separated list to return one value. Instead, it returns separate averages for all of the values in the list. I believe this is on the right track, but I am not sure how to achieve one average from a return-separated list field.