There unfortunately isn't a true dialog box setup - however this can be accomplished easily with form rules or navigation to second form.
In the form rule setup, instead of a button just have them actually check the box and use form rules to show and require the comments accordingly.
If you like the button - I would recommend having the button navigate them to a different Quickbase form that only has that checkbox field and the comments field on it - so basically they're navigating to a workflow form where they can only enter comments. You can use form rules to then check the box or complete the actions that your current button is performing.
