Limit field update to only users in the Custom User Set

We have created a feature to enable Executives from several different organizations to vote on key features they want in our RoadMap. For each org only the named executive, a named proxy or the portfolio administrator should be able to cast the vote.

We have set up Custom User Sets for each organization.

How can we limit that the voting field (and associated comments) can only be updated by one of the individuals in the custom user set?