Hey Michelle,
This is a pretty common issue, and the simple answer is you can only summarize a record value once, either by the primary, secondary OR tertiary user field. What you need is that same "Record" to be represented by user for any and all "assignment types" they might be listed in, and to do that you would want a junction table between the user table, and the record where they're being assigned. (aka a Many-to-Many relationship).
What you accomplish by doing this is making the user, the record, and the "assignment type" a unique record, and your reporting would be done from this new "Assignments" table.
Hope that helps!