Forum Discussion

Bharath's avatar
Bharath
Qrew Member
3 days ago
Solved

Pipeline after import (heavy traffic/transient error)

I am trying to improve pipeline performance and need guidance on the best approach please.

When approx 1000 records are created from csv import using ImportExport action in TABLE A, the pipeline should look for TABLE B ID based on a text field in TABLE A and relate. Attached sample pipeline.

When records are imported, the pipeline runs, but noticed the traffic warning. Bulk upsert may not work as each pipeline trigger is on single record update.

 

Pipelines
  • MarkShnier__You's avatar
    MarkShnier__You
    2 days ago

    Well in fact, there is a new pipeline trigger step that was just released to deal with this situation where you don't want to flood the app with pipeline steps creating too much simultaneous traffic, or another use case is that you want the records processed individually, and not in parallel.

     

     

     

    The new pipeline trigger step I believe is called on bulk event,
    (sorry I'm just walking outside now and I'm on my phone)

     

    so what happens is you do the import and because that is the trigger, the pipeline does not start running until all of the records have been imported. Then there will be a for each loop after that and that will run one by one. Now this is going to mean that it will take much longer for the pipeline to complete, but the advantages you won't get traffic warnings or worse, yet failures, and the other good thing is it won't impact the performance of the app for your regular users using the normal human UI. 

     

    You can test this by rebuilding from scratch or you can copy your existing pipeline, delete the current trigger step and add the on bulk event trigger at the top and edit the copy of your pipeline.

4 Replies

  • Bharath's avatar
    Bharath
    Qrew Member
    2 days ago

    Process time within 15 minutes is acceptable for me. So I am fine with your first suggestion.

    I agree with the emails to include both pipeline name and error message. I had to copy the url and paste it in shared pipeline account.

  • Bharath's avatar
    Bharath
    Qrew Member
    2 days ago

    It worked. Thank you Mark.

    Though it took approx 8 minutes to process 600 records, there is no traffic or performance issue.

     

    • MarkShnier__You's avatar
      MarkShnier__You
      Icon for Qrew Legend rankQrew Legend
      2 days ago

      Thank you for reporting that timing test. So it appears that when the records are run one by one in the for each loop an order of magnitude is they run at a rate a little bit faster than one per second.

       

      So that's kind of slow, but I guess we just need to decide how important speed is in our use case. 

      Just a comment that another good use case for using the bulk of Bulk Event Trigger is when you import say a whole bunch of child records, which may or may not need a parent record created. If you do it the normal away what happens as you get a race condition and you can end up with two orphans, running in parallel asking for the same parent to be created, which of course causes pipeline errors when the second orphan tried to create the same Parent. 

      The process still runs OK but it generates a lot of errors in your email and then what happens of course is when we get too many pipeline errors we ignore them all and don't take the time to investigate the important pipeline errors.

      Maybe one day we will get the Pipeline Name in the error message instead of that horrible 16 digit number. 

  • MarkShnier__You's avatar
    MarkShnier__You
    Icon for Qrew Legend rankQrew Legend
    2 days ago

    Well in fact, there is a new pipeline trigger step that was just released to deal with this situation where you don't want to flood the app with pipeline steps creating too much simultaneous traffic, or another use case is that you want the records processed individually, and not in parallel.

     

     

     

    The new pipeline trigger step I believe is called on bulk event,
    (sorry I'm just walking outside now and I'm on my phone)

     

    so what happens is you do the import and because that is the trigger, the pipeline does not start running until all of the records have been imported. Then there will be a for each loop after that and that will run one by one. Now this is going to mean that it will take much longer for the pipeline to complete, but the advantages you won't get traffic warnings or worse, yet failures, and the other good thing is it won't impact the performance of the app for your regular users using the normal human UI. 

     

    You can test this by rebuilding from scratch or you can copy your existing pipeline, delete the current trigger step and add the on bulk event trigger at the top and edit the copy of your pipeline.