Forum Discussion

MichaelTamoush's avatar
MichaelTamoush
Qrew Captain
3 years ago

Pipelines recently very very slow?

Trying to determine if this is on my end or QBs.

Everything about pipelines has been extraordinarily slow the lat few weeks. Even just opening the page can take me 2-3 minutes for it to load. The pipelines take minutes, not seconds to run. Some take over 5 minutes.

Anyone else experiencing this?



------------------------------
Mike Tamoush
------------------------------
Pipelines

22 Replies

Sort By
  • EvanMartinez's avatar
    EvanMartinez
    Qrew Elite
    3 years ago
    Hi Everyone,

    In the last week we saw an uptick in performance pain and performance cases around Pipelines and wanted to provide some information here in this thread. It can absolutely be frustrating to have your automated workflows delayed and it is something Quickbase takes very seriously when we get that feedback and see cases of customers in pain. Over the last 12 months the usage of Pipelines has grown by over 5,000%, which is great cause it means we have opportunity to pour even more innovation and enhancement into it but has come with some growing pains. The good news I wanted to pass on is that, as we speak, our Product team is working on several major enhancements to increase the performance & execution of Pipelines.

    The biggest improvement is focused on moving all customers to a new behind-the-scenes engine (named "Blaze Engine") that processes the rules and logic of each Pipeline up to 5x faster. We expect to have this new engine rolled out to all customers by our September 2021 product release. For those of you who are still having performance pains right now if you haven't submitted a case yet with our Care team we do recommend you do so as they can look at your specific experience and ideally provide you some steps to help with making your experience better today. There are often things we can do today to help alleviate slowness.

    For those of you working with Pipelines and looking for a little help on Best Practices to set them up with an eye to performance we do have some resources available when building Pipelines, especially when building Loops or handling lots of records in bulk. We recommend reviewing our free on-demand video content from Empower 2021 about Optimizing your Pipelines (9 minutes) and Troubleshooting your Pipelines (40 minutes). When you're trying to do bulk-record operations (something to more than 100 records), you should make sure you use these 4 steps, in this order: 

     A. Prepare Bulk Upsert, B. Search Rows, C. Add Row to Bulk Upsert, and D. Commit Upsert

    This optimizes for the best performance and we are working up a guide that will go over setting this up in more depth right now. Many of these Best Practices will also appear on the "Pipelines Certification" we are launching later this year... We hope this information helps you, and we look forward to continuing to improve your Quickbase experience each and every day. I want to thank everyone for sharing their experience and being open with your frustrations. We want to make sure we can handle not just this massive growth but another 5,000% increase and that is only possible by hearing your feedback and incorporating it into our plans.

    ------------------------------
    Evan Martinez
    Community Marketing Manager
    Quickbase
    ------------------------------
    • MichaelTamoush's avatar
      MichaelTamoush
      Qrew Captain
      3 years ago
      Evan,

      This seems like a boiler plate response with no real solution. They have moved a number of my pipelines to the new Engine, and I see marginal or no improvements. Telling developers to try to build more efficient Pipelines is great, but when a single 'if' statement takes 2-5 minutes to process, it doesn't matter how efficient the pipeline is, it will be so slow it is unusable.

      I could understand if the pipelines were previously taking 5 seconds, and are now taking 45 seconds. The solutions you propose would be fine. But that is not the case. Pipelines that previously took 5 seconds are taking in some cases over an hour!!!

      Even if you achieved 5 times faster that is still 12 minutes+ for one run!

      You are telling us Pipelines has grown too fast for you to keep up with, yet also are forcing everyone to move their automations to Pipelines.

      If other developers are reading this and share my frustration, does anyone have ideas of what we can do to combat this?

      ------------------------------
      Mike Tamoush
      ------------------------------
      • EvanMartinez's avatar
        EvanMartinez
        Qrew Elite
        3 years ago
        Hi Mike,

        I'm sorry that you are still feeling frustrated and that is absolutely understandable given the slowness you are experiencing. You are correct that in some instances there are Pipelines still experiencing slowness right now and that is something that needs to be adresses. I'm hoping it might help if I clarify a bit on the engine and why it might help a little right now but will help more in the long term. It actually would have the advantage long term of not just speeding up individual Pipelines so that they themselves run faster but we are also working to address how many can run concurrently.

        You are probably familiar with the concept that Quickbase apps often exist in server with several other applications and on occasion an application can feel pain when a lot of requests get lined up and have to be processed in order. While the engine that runs Pipelines is more complex than one line it does still have a limit on its queuing and essentially how many Pipelines get attention at once, in the case of our growing environment and high traffic that can result in slowdowns like we are seeing this last week. We aren't just going to get you through the lines faster by speeding up your individual Pipelines we are also working on opening up more lanes so that you feel less impact due to traffic. So the engine itself will help things run more quickly, best practices will help them be more efficient with that time, and then increased capacity and load balancing improvements will help open up access to resources and more smartly share them where the need arises.

        Being able to really focus on this work long term is part of the reason for the Automations end of life program, trying to make both platforms meet their growing demands would have meant not being able to commit more to one without bigger sacrifices in customer experience. Before the announcement since Automations was very familiar to builders its usage was also growing alongside Pipelines meaning both platforms would have needed this kind of performance attention to avoid slowdown and other issues. 

        It is important for us to hear about your frustrations and experiences though to help push those priorities and innovations. When we hear this dissatisfaction we want to make sure we are communicating what we are doing about it even, even when parts of it are more long term.

        ------------------------------
        Evan Martinez
        Community Marketing Manager
        Quickbase
        ------------------------------
  • JeremyAnson's avatar
    JeremyAnson
    Qrew Cadet
    3 years ago
    The My Pipelines page does take longer to load than I would expect, but I'm not sure if it's become slower. Perhaps the inclusion of error stats has slowed it.

    The performance of pipelines in general is also slower than I would expect and makes it difficult/impossible to rely on them to run in a reasonable time. It would be great if they could be used to perform 'back end processing' (i.e. responding to user actions and presenting updated information back to the user) but I guess this isn't their intended purpose.

    ------------------------------
    Jeremy Anson
    ------------------------------
    • MichaelTamoush's avatar
      MichaelTamoush
      Qrew Captain
      3 years ago
      I put in a support ticket and they are looking into it. While I have always experienced pipelines being slower than expected/desired, I think this instance may be my apps being effected by someone else's. I had a moderate to low complex pipeline take nearly 20 minutes to run, so that tells me something is off.


      ------------------------------
      Mike Tamoush
      ------------------------------
  • AustinK's avatar
    AustinK
    Qrew Commander
    3 years ago
    How many pipelines do you have in total? Loading into our main account for pipelines I would say it takes a good 15 seconds for everything to fully load. But we only have 30 pipelines and not all of them are running.
    • MichaelTamoush's avatar
      MichaelTamoush
      Qrew Captain
      3 years ago
      I probably have around 75 pipelines, and once i import all my automations will have about 175.......

      But right now takes about 3 minutes to load so i am assuming itll be 10 minute load time once i import. I am not loving this....

      But the bigger issue is pipeline run time. The pipelines that used to take 15 seconds to run are taking 15 minutes now.

      ------------------------------
      Mike Tamoush
      ------------------------------
      • AustinK's avatar
        AustinK
        Qrew Commander
        3 years ago
        I am now noticing the slowness as well. For example I have a Pipeline that updates 2 tables in one app based on 2 tables in another app. There are 11 records in table A and 141 records in table B and it takes 5 minutes to grab ~150 records that have very little data to move the few that have changed to another table. 

        I did this with a similar pipeline from the same 2 tables but to another app that was used much more and it took even longer than the first. So maybe it is queuing up the pipeline changes with other actions inside of the apps?