Pipeline Design for Bulk Record Updates
Hi all. I have a timecards table where users track all the things they're working on. Relatedly, we have a separate child timecard table, with a narrower field list but otherwise the exact same recor...
Forum Discussion
interesting, thanks Mark. Sounds like I misunderstood the purpose of the bulk event step. So for updating thousands of records all at once, it's still easiest/most performative to just have the pipeline loop through all the records one at a time (outside of leveraging Sharon's self-created methodology)?
QuickBaseJunkie
Qrew Legend
Qrew LegendHarrisonSmith​ This is the demo Mark was referring to during Empower Pro https://www.quickbasejunkie.com/blog/empowerpro24 it would be Step C: Make Request - using the API_ImportFromCSV API that would be the most interest to you. Only in place of referencing the json output you would be referencing the step A bulk trigger (where you deleted the loop). Such as
{%- for i in a %}
and
{{i.id}},{{i.start_time}}...
Quickbase is working on a "Bulk Import" step to go with the "Bulk Trigger". The beta for it has just started. So if this isn't urgent it may be worth waiting for it to be released.
-Sharon
- MarkShnier__You
Just one more thought about the bulk trigger pipeline. If you want a compromise between the regular on new event, which floods your App with 5,000 requests at the same time and the Bulk Trigger which processes one by one, you could speed up the Bulk trigger by a factor of two by making a field which determines if the Record ID is even or odd.
Then you could duplicate the Bulk Trigger pipeline and had one of them do the evens and one of them do the odds. The effect of that is you would have the records being processed twice as fast through the Bulk Trigger Pipelines, but that still may be slow enough to have no impact on the performance of your app for the regular users.
Interesting. Thank you both. I'll watch Sharon's demo and see what I can come up with.
