There are two ways you can handle the returning to the correct parent form. One is encoding the correct dfid in the URL like Matthew suggested. It would be a formula url like :
URLRoot() & "db/ABC123456?a=er&dfid=12_fid_9=" & URLEncode ([Related Parent Record ID#]])
ABC123456 = parent table dbid
This formula will only take you to the specific form (form id 12). You can use some logic to dynamically change the form id it goes back to or use multiple buttons which take you to the different forms. This formula field will be in the child table
second way is - to encode "&z=" & Rurl() at the end of your add child record formula URL in the parent table.
the z=rurl() will return to the prior page (be it a form or a report). That means once the user is done adding the child record he/she will return to the original form/page they started with.
An example would be URLRoot() & "db/DEF123456?a=API_GenAddRecordForm&_fid_9=" & URLEncode ([Parent Record ID#])& "&z=" & Rurl()
the DEF123456 = child table dbid.
