Use Case: Master Table vs Pipeline?

• Do you happen to have a comprehensive article on this topic? There is a collection here.  But it's pretty scattered.  The native help text,  imho,  still needs a lot of work.   https://community.quickbase.com/blogs/bree-mackey1/2020/07/22/so-you-want-to-learn-pipelines
• You mention needing to build a 'temporary' external box that contains the columns and rows and search for the criteria to fill the external box (inclusive of the related project field).   
  • Could the temporary external box be an Excel table?   NO.  I was being perhaps too "glib".  The Quickbase Channel has a twisty to open for Bulk Record Sets.  There is an option there to create the Bulk Upsert (ie the big old empty cardboard box) 
  • Curious why this has to be external to QB? (I'm I'm just saying that if you need to create 50 records in QuickBase you could do it by searching your Master table of tasks and then on the For Each step you could one by one create those 50 child Tasks.  That will certainly work but you are hammering 50 APIs to your app all at once to create the 50 records and in theory it will slow down your app while those are being created. Now 50 records is not going to kill you but it's just much better practice to let the pipeline do more of the work. The pipeline runs on a different engine than your Quickbase app. The QuickBase app runs under the proprietary database of QuickBase and can only do one thing at a time.  (ie while you 50 records are being added one by one, someone who is waiting for a dashboard to populate will be waiting.  Pipelines runs in a cloud, I forget if it is a AWS or a different cloud service.   But it effectively has unlimited capacity. So it is better to query your master table once which is very quick and then have the pipeline do the work of filling the cardboard box and then uploading all those records all at once. That way you are only hitting on your app once to do the query and once again to do the upload so that is a lot less impactful on performance than hitting your app 50 times.  
  • Could the source instead be the internal Task Master Table which was constructed for this purpose?  I am not sure what you mean by this question. The pipeline will query the Master Task table in order to find the records to put in the Bulk Upsert.   Just by the way what the word "Upsert"  means is that it will Insert new records or UPdate old records. In your case they will be all inserting new records but there could be a different use case where you are merging data in where there is a key field and in that case the upsert will update existing records and insert new ones as required. This would be very similar to how a native Quickbase table to table copy works or manual import of an Excel sheet into a Quickbase table where there is a key field.  In those cases it will either update existing records or create new ones.     
• I presume the dragging of fields is a one-time set up step.   Please confirm.  Yes you are building the pipeline once. 
• I presuming the step of opening a new tab to watch the Pipeline at work is a testing step that at any point could be utilized for troubleshooting if needed.  Please confirm.  Yes, this is just a debug step that you would normally do after building or editing any pipeline trigger the pipelines and then watch the recap to confirm it runs OK.  You don't have to watch it live, as there is a button to look at the activity for a pipeline but I think it's just much less confusing to be watching when it is triggered.

Mike, exactly. 

The other thing you will observe if you watch the pipeline run if you try to add the  records individually is that the For Each loop fires asynchronously. What that means is basically it hammers the application with 50 API requests all at the same time. Because QuickBase cannot do 50 things at once it rejects many of the API requests. Now, fortunately the pipeline re-Queues those requests automatically and then fires the same request after a short delay.  

I think I recall seeing an indication that it was willing to try the same request up to 15 times before it gives up.

But imagine a use case where you were adding say 2000 records. So you're app gets hammered with 2000 instantaneous API calls and I think that there's a risk that some of them could timeout because every time it goes to retry a rejected request the app would still be busy processing others of those 2000 API's.

Also, if you had 2000 individual request I think your users would definitely feel like the app became unresponsive.

So sure, in practice, probably 50 individual requests you could get away with easily and nobody would really notice but it's just basic good practice if you are going to add dozens of records to take the extra time to create that bulk upsert and do it "right ". One API call to Query and one API call to Upsert.