IsNull use with a numeric ratings field

I'm trying to do a clumsy average across multiple numeric fields by summing the fields and then dividing by a simple sum of ToNumber(Not(IsNull())) of those fields (effectively COUNTIF not blank). 

Problem is my numeric rating field (0-5), when no value is present, is showing up as not blank. 

Any thoughts on this?

To summarize:

Numeric ratings field, values are: blank, 0, 1, 2, 3, 4, 5 (default for a numeric rating field)
Formula numeric field: ToNumber(not(isnull([default numeric field name]))) that comes up as saying 1 (is not blank) when the blank choice is chosen.

Thanks for your thoughts in advance!