One way to do this:
Say the [Payee] field is located in the "Payments" table.
You'll need a relationship from your Vendors table to the Payments table - "Vendors" would be the parent table.
Relationship: 1 Vendors has many Payments
From that Vendors table - you'll have the [Related Vendor] field, and a reference proxy field such as [Vendor Name].
So on the Payments form - you add a dynamic form rule
If [Payee] is equal to the value "Vendor"
Show [Vendor Name].
This displays the [Vendor Name] dropdown when [Payee] = "Vendor", otherwise it hides it.
So if [Payee] = "Client" - you're done. [Vendor Name] remains blank.
If [Payee] = "Vendor", then you can select from the [Vendor Name] dropdown.