Are you using an "Average of" summary field? This should exclude empty fields from the computation like you want.
I create three records with a number field and an "Average of Number" summary field.
Average = 2 as expected
clear the 2 from the second record and
Average = 2, blank entry automatically excluded.
Average = 1.33333, zero is not excluded
human error (as usual). in one of my calculations - i found that if the field is empty; 0.
that screwed everything up.