Discussions

 View Only
Expand all | Collapse all

Pipelines recently very very slow?

  • 1.  Pipelines recently very very slow?

    Posted 07-27-2021 16:24
    Edited by Michael Tamoush 07-27-2021 16:37

    Trying to determine if this is on my end or QBs.

    Everything about pipelines has been extraordinarily slow the lat few weeks. Even just opening the page can take me 2-3 minutes for it to load. The pipelines take minutes, not seconds to run. Some take over 5 minutes.

    Anyone else experiencing this?



    ------------------------------
    Mike Tamoush
    ------------------------------


  • 2.  RE: Pipelines recently very very slow?

    Posted 07-28-2021 03:18
    The My Pipelines page does take longer to load than I would expect, but I'm not sure if it's become slower. Perhaps the inclusion of error stats has slowed it.

    The performance of pipelines in general is also slower than I would expect and makes it difficult/impossible to rely on them to run in a reasonable time. It would be great if they could be used to perform 'back end processing' (i.e. responding to user actions and presenting updated information back to the user) but I guess this isn't their intended purpose.

    ------------------------------
    Jeremy Anson
    ------------------------------



  • 3.  RE: Pipelines recently very very slow?

    Posted 07-28-2021 10:23
    I put in a support ticket and they are looking into it. While I have always experienced pipelines being slower than expected/desired, I think this instance may be my apps being effected by someone else's. I had a moderate to low complex pipeline take nearly 20 minutes to run, so that tells me something is off.


    ------------------------------
    Mike Tamoush
    ------------------------------



  • 4.  RE: Pipelines recently very very slow?

    Posted 07-28-2021 10:26
    How many pipelines do you have in total? Loading into our main account for pipelines I would say it takes a good 15 seconds for everything to fully load. But we only have 30 pipelines and not all of them are running.


  • 5.  RE: Pipelines recently very very slow?

    Posted 07-28-2021 11:34
    I probably have around 75 pipelines, and once i import all my automations will have about 175.......

    But right now takes about 3 minutes to load so i am assuming itll be 10 minute load time once i import. I am not loving this....

    But the bigger issue is pipeline run time. The pipelines that used to take 15 seconds to run are taking 15 minutes now.

    ------------------------------
    Mike Tamoush
    ------------------------------



  • 6.  RE: Pipelines recently very very slow?

    Posted 07-30-2021 16:28
    I am now noticing the slowness as well. For example I have a Pipeline that updates 2 tables in one app based on 2 tables in another app. There are 11 records in table A and 141 records in table B and it takes 5 minutes to grab ~150 records that have very little data to move the few that have changed to another table. 

    I did this with a similar pipeline from the same 2 tables but to another app that was used much more and it took even longer than the first. So maybe it is queuing up the pipeline changes with other actions inside of the apps?


  • 7.  RE: Pipelines recently very very slow?

    Posted 07-30-2021 19:59

    Austin,

    If you could submit a support ticket that might help. Last time this happened, I fought with customer support and they were doing nothing until a number of other reports came in, and they found out it was on their side.

    They are looking into mine right now, but all my Pipelines that used to run in 7-10 seconds are taking 3-7 minutes to run. Some up to 13 minutes.

    You can check how long any pipeline is taking by clicking on 'Activities' then filtering by 'Start/Finish'. You can scroll through and check the run times now verse a week ago. Would be curious what you find. Something is definitely very wrong, but I don't think customer support is really listening right now because they probably think it's isolated to me and that I have inefficient Pipelines or something. While i do have some poorly written pipelines, the fact that all of my pipelines have gone from running in 20 seconds or less to 3+ minutes and up to 13 minutes.....something is wrong.



    ------------------------------
    Mike Tamoush
    ------------------------------



  • 8.  RE: Pipelines recently very very slow?

    Posted 08-02-2021 18:51
    Hi Mike. Any news regarding your issue? My pipelines are taking forever now too. I have 2 pipelines, each has about 3-4 actions. It takes about 10 minutes to run these small pipelines

    ------------------------------
    Jason Bini
    ------------------------------



  • 9.  RE: Pipelines recently very very slow?

    Posted 08-02-2021 20:54
    All they tell me is 'we escalated it to the Pipelines team', but its been a week. Submit a ticket as well. If they get blasted with complaints they will take it seriously.

    ------------------------------
    Mike Tamoush
    ------------------------------



  • 10.  RE: Pipelines recently very very slow?

    Posted 08-03-2021 09:00
    yes sir ticket is opened.

    ------------------------------
    Jason Bini
    ------------------------------



  • 11.  RE: Pipelines recently very very slow?

    Posted 08-04-2021 09:46
    FYI.. my pipeline is doing the same thing, running VERY slow. It used to run in 1-3 minutes and is now taking up to 60 mins. You can see every step takes 2-5 minutes to even kick off, even simple steps like an "IF" condition. Very frustrating and I have an open ticket with the team. No response yet.

    ------------------------------
    Robert
    ------------------------------



  • 12.  RE: Pipelines recently very very slow?

    Posted 08-04-2021 10:01
    I can't even build a Pipeline now because trying to test it is impossible when it takes multiple minutes to test even an if condition.

    I say keep bombarding customer support every day. It has been like this for at least 2 weeks. They told us they are phasing out automations, so no point in building those, but Pipelines doesn't even work right now.

    I am beyond frustrated.

    ------------------------------
    Mike Tamoush
    ------------------------------



  • 13.  RE: Pipelines recently very very slow?

    Posted 08-04-2021 10:11
    I had a thought. Since we cannot build Automations as they told us they are phasing them out, and Pipelines is currently unusable, this would equate to QuickBase not delivering the product as they have promised. Perhaps if we as developers ask for credits (ie money back) for every day Pipelines is unusable (I cannot build one at this point, it is too hard to test because of the lengthy wait times), perhaps this will be escalated to the highest priority?

    ------------------------------
    Mike Tamoush
    ------------------------------



  • 14.  RE: Pipelines recently very very slow?

    Posted 08-06-2021 11:38
    Hi Everyone,

    In the last week we saw an uptick in performance pain and performance cases around Pipelines and wanted to provide some information here in this thread. It can absolutely be frustrating to have your automated workflows delayed and it is something Quickbase takes very seriously when we get that feedback and see cases of customers in pain. Over the last 12 months the usage of Pipelines has grown by over 5,000%, which is great cause it means we have opportunity to pour even more innovation and enhancement into it but has come with some growing pains. The good news I wanted to pass on is that, as we speak, our Product team is working on several major enhancements to increase the performance & execution of Pipelines.

    The biggest improvement is focused on moving all customers to a new behind-the-scenes engine (named "Blaze Engine") that processes the rules and logic of each Pipeline up to 5x faster. We expect to have this new engine rolled out to all customers by our September 2021 product release. For those of you who are still having performance pains right now if you haven't submitted a case yet with our Care team we do recommend you do so as they can look at your specific experience and ideally provide you some steps to help with making your experience better today. There are often things we can do today to help alleviate slowness.

    For those of you working with Pipelines and looking for a little help on Best Practices to set them up with an eye to performance we do have some resources available when building Pipelines, especially when building Loops or handling lots of records in bulk. We recommend reviewing our free on-demand video content from Empower 2021 about Optimizing your Pipelines (9 minutes) and Troubleshooting your Pipelines (40 minutes). When you're trying to do bulk-record operations (something to more than 100 records), you should make sure you use these 4 steps, in this order: 

     A. Prepare Bulk UpsertB. Search RowsC. Add Row to Bulk Upsert, and D. Commit Upsert

    This optimizes for the best performance and we are working up a guide that will go over setting this up in more depth right now. Many of these Best Practices will also appear on the "Pipelines Certification" we are launching later this year... We hope this information helps you, and we look forward to continuing to improve your Quickbase experience each and every day. I want to thank everyone for sharing their experience and being open with your frustrations. We want to make sure we can handle not just this massive growth but another 5,000% increase and that is only possible by hearing your feedback and incorporating it into our plans.

    ------------------------------
    Evan Martinez
    Community Marketing Manager
    Quickbase
    ------------------------------



  • 15.  RE: Pipelines recently very very slow?

    Posted 08-06-2021 13:08
    Evan,

    This seems like a boiler plate response with no real solution. They have moved a number of my pipelines to the new Engine, and I see marginal or no improvements. Telling developers to try to build more efficient Pipelines is great, but when a single 'if' statement takes 2-5 minutes to process, it doesn't matter how efficient the pipeline is, it will be so slow it is unusable.

    I could understand if the pipelines were previously taking 5 seconds, and are now taking 45 seconds. The solutions you propose would be fine. But that is not the case. Pipelines that previously took 5 seconds are taking in some cases over an hour!!!

    Even if you achieved 5 times faster that is still 12 minutes+ for one run!

    You are telling us Pipelines has grown too fast for you to keep up with, yet also are forcing everyone to move their automations to Pipelines.

    If other developers are reading this and share my frustration, does anyone have ideas of what we can do to combat this?

    ------------------------------
    Mike Tamoush
    ------------------------------



  • 16.  RE: Pipelines recently very very slow?

    Posted 08-06-2021 18:34
    Hi Mike,

    I'm sorry that you are still feeling frustrated and that is absolutely understandable given the slowness you are experiencing. You are correct that in some instances there are Pipelines still experiencing slowness right now and that is something that needs to be adresses. I'm hoping it might help if I clarify a bit on the engine and why it might help a little right now but will help more in the long term. It actually would have the advantage long term of not just speeding up individual Pipelines so that they themselves run faster but we are also working to address how many can run concurrently.

    You are probably familiar with the concept that Quickbase apps often exist in server with several other applications and on occasion an application can feel pain when a lot of requests get lined up and have to be processed in order. While the engine that runs Pipelines is more complex than one line it does still have a limit on its queuing and essentially how many Pipelines get attention at once, in the case of our growing environment and high traffic that can result in slowdowns like we are seeing this last week. We aren't just going to get you through the lines faster by speeding up your individual Pipelines we are also working on opening up more lanes so that you feel less impact due to traffic. So the engine itself will help things run more quickly, best practices will help them be more efficient with that time, and then increased capacity and load balancing improvements will help open up access to resources and more smartly share them where the need arises.

    Being able to really focus on this work long term is part of the reason for the Automations end of life program, trying to make both platforms meet their growing demands would have meant not being able to commit more to one without bigger sacrifices in customer experience. Before the announcement since Automations was very familiar to builders its usage was also growing alongside Pipelines meaning both platforms would have needed this kind of performance attention to avoid slowdown and other issues. 

    It is important for us to hear about your frustrations and experiences though to help push those priorities and innovations. When we hear this dissatisfaction we want to make sure we are communicating what we are doing about it even, even when parts of it are more long term.

    ------------------------------
    Evan Martinez
    Community Marketing Manager
    Quickbase
    ------------------------------



  • 17.  RE: Pipelines recently very very slow?

    Posted 08-09-2021 10:19
    Hi Evan,

    Thanks for the detailed update and explanation. 

    For my pipeline, the move to the new engine resulted in great performance. I am seeing it run in 12-15 seconds which is fantastic. Thanks for the continued focus on performance as the usage grows. 

    Robert

    ------------------------------
    Robert
    ------------------------------



  • 18.  RE: Pipelines recently very very slow?

    Posted 08-09-2021 10:30
    Evan,

    Thank you fr the detailed response. I understand the limitation and the need to open more lanes for true speed increase. I think the main frustration is - if you don't have enough lanes to support everyone on Pipelines, then it does not seem logical to force everyone off automation and into Pipelines. It would seem to me you would want to first, achieve enough lanes and speed to support everyone, and THEN tell everyone to move off of Pipelines.

    I think we all can imagine that since you told everyone to move their automations, and then 2 weeks later Pipelines is essentially down, that perhaps too many people migrated over.....

    ------------------------------
    Mike Tamoush
    ------------------------------



  • 19.  RE: Pipelines recently very very slow?

    Posted 08-09-2021 11:14
    Well, after 3 weeks of headaches, frustration, and being told essentially 'there is nothing we can do', boom, this morning everything is running at typical speeds.

    No clue what happened..?! Hopefully this isn't a fluke and it doesn't slow down again tomorrow.

    Curious if platform wide it is better? Anyone else still experiencing issues?

    ------------------------------
    Mike Tamoush
    ------------------------------



  • 20.  RE: Pipelines recently very very slow?

    Posted 10-07-2021 13:46
    I'm not sure if this is platform wide, but I've just watched 3 actions complete in a single pipeline in 30 minutes. Is the best next step to raise a customer support case?

    ------------------------------
    Jeremy Anson
    ------------------------------



  • 21.  RE: Pipelines recently very very slow?

    Posted 10-07-2021 13:53
    Definitely, put in a support ticket.

    ------------------------------
    Mark Shnier (YQC)
    mark.shnier@gmail.com
    ------------------------------



  • 22.  RE: Pipelines recently very very slow?

    Posted 10-07-2021 15:18
    Agree with Mark. Put in a ticket. I have not had problems since the fixed whatever the issue was. One good thing to check:

    View activities of a pipeline, and then in the dropdown selection up top choose start/finish. This will show the start finish times of the pipeline. You can check multiple pipelines and see if only one is having issues, or all of them. When I had trouble, ALL my pipelines were taking 30-90 minutes to run.

    If it is just one pipeline, it's more likely there is some search loop or inefficient design causing the trouble.

    ------------------------------
    Michael Tamoush
    ------------------------------



  • 23.  RE: Pipelines recently very very slow?

    Posted 10-08-2021 03:05
    Thanks for the advice.

    ------------------------------
    Jeremy Anson
    ------------------------------