I have a formula in one of my Quick Bases that looks like this...
ToTimestamp([Stop Date],[Stop Time])-ToTimestamp([Start Date],[Start Time])
It is similar to yours only it includes the date as well, perhaps it will also resolve your issue.
I've also found that using 'smart units' on the duration field can cause issues. If you have that selected, you can change it to minutes and see if that helps.
To 'add' multiple durations I've set them us as child records and summarized the duration on the parent record without issue.
-Sharon