New Form Button display settings

I don't think you can control that. I think that the font colour depends on the colour of the button. For example you can imagine if you had a yellow button then you would not want to see white letters because they would not be readable. Conversely if you had a very dark red button then black letters would be hard to see. So they have some built-in logic under the covers to have the font colour be contrasting with the button color. I can't really explain why one button has a black border and one does not.

The alternative is to invest the time to make your own rich text formula buttons and then you can control everything but there's more set up involved with that so may not be worth the effort.

Here is an example of the code that you would use for a rich text formula button.  

var text ButtonColor = "blue"; //use hex code but most color names work

var text WordsColor = "white"; //typically this will be white or black 

var text URL = [your URL field here];

var text Words = "Break"; // use a formula 

// Begin button style
var text bgcolor = $ButtonColor;
var text txtcolor = $WordsColor;
var text style = "style=\"text-decoration: none; text-align: center; background:" & $bgcolor & "; "

&  " width: 50px; " // width is optional otherwise will be as wide as the words

&  " border-radius: 5px; padding: 6px 10px; color: " & $txtcolor & "; display: inline-block; font: normal 700 24px/1 \"Calibri\", sans-serif; text-shadow: none;";
// End button style    
  
 "<a " & $style & " href=" & $URL & ">" & $Words & "</a>"