Pipeline Design for Bulk Record Updates
Hi all. I have a timecards table where users track all the things they're working on. Relatedly, we have a separate child timecard table, with a narrower field list but otherwise the exact same recor...
Forum Discussion
MarkShnier__You
Qrew Legend
Qrew LegendThe purpose of the On New Bulk event is to ensure that records are processed one by one and don't overwhelm the system and also in use cases where you really want the first record processed before you start processing the second. A small example of that would be where child records are being created in bulk and sometimes they need a corresponding parent record created. If you do not use Bulk Trigger you will get multiple pipelines running simultaneously and both trying to create the same unique parent record. One will fail and generate errors.
But you are right, in terms of performance, Bulk Triggers is going to make the process run slower.
If this is important to you and you can scrounge up some budget for outside consulting, I'm pretty sure that Sharon Faust (Quickbase Junkie) can help you.
A while back Sharon did a Qrew Demo where she used advanced techniques in Pipelines with Jinja which allowed her to achieve a 10x or maybe 100x speed up in processing speed.
interesting, thanks Mark. Sounds like I misunderstood the purpose of the bulk event step. So for updating thousands of records all at once, it's still easiest/most performative to just have the pipeline loop through all the records one at a time (outside of leveraging Sharon's self-created methodology)?
- MarkShnier__You
.... Yes, aside from Sharon, you should be using the Bulk Step and maybe you don't really care if it takes hours and hours to run because your app will still "Performative" (running nicely) for your regular users.
A very nice design feature of the Bulk Trigger and the Regular Trigger is you can set the number of records for it to trigger on.
So for example, you can set the Regular one to trigger when a maximum of 1 record is updated or created, and that will ensure that your manual entry users still get instant processing. And then you set the Bulk Trigger one to to trigger at 2 or more records being changed.
When I went looking for that setting on the regular trigger for On New Event, in an older Pipeline, the setting at the bottom of the box for the maximum number of records to trigger on, was not there. I was initially confused so I started to build a brand new pipeline and that same On New Event trigger did have the new setting at the bottom of the box. So you may have to go back to your existing regular On New Event trigger and replace it the same so that it will have that new feature.
- QuickBaseJunkie
HarrisonSmith​ This is the demo Mark was referring to during Empower Pro https://www.quickbasejunkie.com/blog/empowerpro24 it would be Step C: Make Request - using the API_ImportFromCSV API that would be the most interest to you. Only in place of referencing the json output you would be referencing the step A bulk trigger (where you deleted the loop). Such as
{%- for i in a %}
and
{{i.id}},{{i.start_time}}...
Quickbase is working on a "Bulk Import" step to go with the "Bulk Trigger". The beta for it has just started. So if this isn't urgent it may be worth waiting for it to be released.
-Sharon- MarkShnier__You
Just one more thought about the bulk trigger pipeline. If you want a compromise between the regular on new event, which floods your App with 5,000 requests at the same time and the Bulk Trigger which processes one by one, you could speed up the Bulk trigger by a factor of two by making a field which determines if the Record ID is even or odd.
Then you could duplicate the Bulk Trigger pipeline and had one of them do the evens and one of them do the odds. The effect of that is you would have the records being processed twice as fast through the Bulk Trigger Pipelines, but that still may be slow enough to have no impact on the performance of your app for the regular users.
