I am new to Pipelines and also new to Master Tables and would like community guidance:Use Case: We have a complex Project Management Tracking System built in QuickBase. (400 projects, 150 users, 40 of which are Project Managers)
What is the best method to automatically create the 50 new task records in the Tasks table each time a New Project Request has been completed? In QuickBase documentation, there is an article on Master Tables, but before I go that route, I am wondering if perhaps a Pipeline is a better way to solve this Use Case. I am new to Pipelines and taking some of the QB University Training and am confused about things such as 'Bulk Record Upsert' and other options.Please provide general guidance on the best way to proceed.Thanks in advance for your guidance.
Mark -Thanks (as always) for your prompt and informative reply. Thanks for confirming this would be a great Use Case for pipelines. I will admit that I have not yet discovered a comprehensive article on Pipelines so struggling to get a good definition of options such as 'upserts'. Could you please answer a few more questions to help me get my head around this concept?
Mark,Thanks for the great explanation on Bulk Upsert. I've been trying to understand what it is and when to use it.So do I understand correctly:This WOULD work by simply doing a search in the master table, and the For Each: Create the child record. But that would take system resources as 50 create APIs will run?
However, if I do the Bulk Upsert technique, in the For Each loop it only performs one API call?
The other thing you will observe if you watch the pipeline run if you try to add the records individually is that the For Each loop fires asynchronously. What that means is basically it hammers the application with 50 API requests all at the same time. Because QuickBase cannot do 50 things at once it rejects many of the API requests. Now, fortunately the pipeline re-Queues those requests automatically and then fires the same request after a short delay.
I think I recall seeing an indication that it was willing to try the same request up to 15 times before it gives up.But imagine a use case where you were adding say 2000 records. So you're app gets hammered with 2000 instantaneous API calls and I think that there's a risk that some of them could timeout because every time it goes to retry a rejected request the app would still be busy processing others of those 2000 API's.Also, if you had 2000 individual request I think your users would definitely feel like the app became unresponsive.So sure, in practice, probably 50 individual requests you could get away with easily and nobody would really notice but it's just basic good practice if you are going to add dozens of records to take the extra time to create that bulk upsert and do it "right ". One API call to Query and one API call to Upsert.