Pipeline to write to Google Spreadsheet triggered by spreadsheet import

Question

So I've been testing writing QB to a Google Sheet whenever a record gets created. Tried testing when a spreadsheet is imported as that is how our users create records, but the pipeline triggers everytime each record is created which can sometimes create a bottleneck and some of the data not being transfered.

In summary: ~100 excel lines imported - > ~100 records created all at once > Pipeline to transfer to Google gets triggered that many times within seconds -> bottleneck ->error -> Google Sheets missing some lines

Is there a way to trigger the pipeline only once when a spreadsheet is created and loop through the records imported one by one? Or could I keep the first model and incorporate the Clock channel to ensure each one finished writing before the pipeline is called again somehow?

Thanks

gesco_qbadminis · Answer

I think that it's somewhere on the Pipelines roadmap to have what is called a bulk trigger. That would allow you to trigger some action after a grid edit or an import, anyways that's my understanding.

But meanwhile, as they say, we are where we are, so we don't let the things that we can't do stop us from doing the things we can.

As it happened I was just reviewing an app I did about six months ago and I had the same problem. Here's how I solved it.

I created a admin record with the Record ID of 1 and linked it to all my detail records.

I created a checkbox flag field on the Parent Admin record to indicate that children had been created but had not yet been imported to the next step of the process, in your case that's the google sheets process.

A pipeline triggers whenever a child was created and it updates this flag to checked. Now of course that pipeline will have fired say 100 times in quick succession but the checkbox would only get checked once, and the other 99 times the pipeline would I have no effect because it would set the checkbox to be checked when it was already checked.

I set up a pipeline to watch for the admin checkbox getting flagged and then I use the Clock step in Pipelines to put in a delay of say 15 seconds. That will give plenty of time for the Child records to all get created in QuickBase.

Then after those 15 seconds the pipeline moves on to process that batch and the last step in the pipeline is to reset the admin checkbox back to unchecked, and hence waiting for the next child record(s) to get created sometime in the future.

markshnier__you · Answer

... I wanted to mention that it was me that did that post above. I was logged into an Client's admin user ID to do some pipelines work, and forgot to change my sign in.&nbsp;

markshnier__you · Answer

Yes, this is a table which one single record in it, so it will be [Record ID#] = 1. Then in your details table you make a formula field called perhaps [Link to Import Admin] with a formula of 1.

Now when your pipeline does run somehow it has to know which of those detail records to import. Although I guess once the pipeline moves the record up to Google sheets than the pipeline can flag the Quickbase record as being imported and therefore not eligible to be imported again on the next batch.

jamesdalton · Answer

Thanks for your response, I will try this... Just for my understanding, will I need to create a new table soley for this admin record with the checkbox?&nbsp;&nbsp;

jamesdalton · Answer

Thanks for your response, I will try this... Just for my understanding, will I need to create a new table soley for this admin record with the checkbox?&nbsp;&nbsp;

Forum Discussion

Pipeline to write to Google Spreadsheet triggered by spreadsheet import