How do a set a formula text field to require a unique value?

you could try this

make a formula which calcuates the name of the file.

ToText([my file attachment field]) and then make a form rule to copy that to a text field all the time.

ie when Formula of file attachment field <> text value of file attachment field change the text to the formula and uncheck the checkbox so the rule always fires.

Then mark the text field unique.