In a formula QuickBase will take the first hit it gets and do that action(checking the box or whatever.) So if the formula will match XField but you want it to be unchecked if YField is also matched in the same one it will not work. Whatever is closer to the top that first matches is what will work.
I might be misunderstanding but it seems like you have a formula checkbox that contains either multiple formulas or one large one that has multiple conditions that overlap. I am leaning more towards the multiple formulas(for both X and Y field) in one checkbox, which should not work based on what I said above.
I understand your formula might be large but it might help others help you if you post it.