Forum Discussion

PaulPeterson1's avatar
PaulPeterson1
Qrew Assistant Captain
3 years ago

Pipeline GUI Issues

I am wondering if anyone else is encountering troubles creating pipelines using the GUI.  I spent most of the day yesterday attempting to create a very simple pipeline that is triggered by a date change and updates that change in another table.  I had a similar Automation that took less than 5 minutes to create.

The overall layout of the pipeline:

Step A - Triggered when the date changes

Step B - change a field in the same table that will be used in the search (as I'm typing this a thought occurred, is the prev option available in the advanced query?  for {{fid.XEX.a.$prev.field_name}}?  I know the syntax is probably wrong, but will that work conceptually)

Step C - Prepare Bulk Record Upsert - All three fields needed for the upsert are defined

Step D - Search for changed records (even though in most cases the search will only return 1 record)

In Do Loop:

Step E - Add a Bulk Upsert Row 

Step F - Update Record in Step A to clear the search flag

Loop Complete

Step G - Commit Upsert

The problems I encountered were: 

A)  Each time I something to a step (trigger field, fields for future use, table, etc.) I get a spinning circle of dots for at least a minute and cannot edit anything while waiting for the precious change to update.

B)  Table, fields, or other required data IS entered and showing in the step but I still get the yellow caution when the step is minimized stating that the step is missing required fields.  Most of the time this will go away if I refresh the schema (IF this works, I have to refresh the Schema several times).  Other times the caution will only go away after refreshing the browser window.

C)  The BIGGEST issue, I am not able to add the fields to the Add a Bulk Upsert Row.  They are defined in Step C - Prepare Bulk Record Upsert.  When I add the step to Add a Bulk Upsert Row I get two error messages.  The first, and most annoying one, just says "Oops, something went wrong."  That has to be the most frustrating and useless error message I have ever seen.  When I refresh the schema I get, "Internal error while refreshing schemas.  Please contact support if you need help"  This message is just as useless.  I have contacted support and after waiting an entire day the issue is no closer to resolution.

Thank you for your patience if you made it this far into my rant.

Is anyone else experiencing the same issues?  If so, how did you resolve them?

@Evan Martinez - Are there any plans to improve the error message in the pipeline GUI?  My experience with them in this case and other instances has been unpleasant.  Are there any plans to make the error messages less vague and ambiguous?  I know error handling can get complicated, but what purpose does "Oops, something went wrong" serve?  Also, are there plans to improve the performance of the pipeline GUI.  I was extremely excited when pipelines were introduced, but have become much less of a fan after trying to creating them.

​

------------------------------
Paul Peterson
------------------------------
Pipelines
  • JoelHickok's avatar
    JoelHickok
    Qrew Cadet
    3 years ago
    Hi Paul,

    I feel your frustration, and cannot provide any specific solutions to your problems right now.  However, I will echo that my experience using Pipelines has been a double-edged sword.

    Pipelines have introduced many powerful features and functions for our organization.  Some of these we already feel are indispensable.  But the User Interface and Pipeline development experience is miserable.  I am shocked that the Quickbase Team has allowed this product to exist for so long now without at least resolving some of the biggest issues.  And now Automations are already on the path for deprecation, while Pipelines don't feel honed in perfectly yet.

    One of my biggest complaints is that you can not re-order steps in the UI, and you cannot insert a brand new trigger as the first step to replace the existing trigger.  You can download the YAML file, carefully edit it, and then create a new Pipeline using that.

    ------------------------------
    Joel Hickok
    ------------------------------
    • MikeTamoush's avatar
      MikeTamoush
      Qrew Commander
      3 years ago

      I'm just here to echo everything Joel said, in hopes QB Development takes notice.

      The user interface can be quite miserable, specifically not being able to reorder steps. Also, if you change the name of a field that you use in Jinja code, you can break your pipeline (it will continue to work unless you go to edit the pipeline in any way, at which point it will break).

      I get a number of error messages etc like you, and like Joel said, the fact they are removing automations before they really fix the issues is disappointing.



      ------------------------------
      Mike Tamoush
      ------------------------------
    • AustinK's avatar
      AustinK
      Qrew Commander
      3 years ago
      You can re-order steps in the UI. Drag and drop but it doesn't always work exactly as you want. However your point stands about the trigger not being able to be replaced easily if at all in some cases through the UI. I only recently learned you could drag and drop in here lol.

      My biggest complaint about Pipelines is the fact that it hides so much from you and it also seems to work based on the field name and not the field ID which seems insane to me. If you have a field that has a name change the Pipeline breaks. Not only does it break but a schema refresh won't fix your issue. You need to go to the original, and possibly all, steps the field is used in and click into the box with the field list. This is what updates the step to use the new schema after it gets refreshed, at least as far as the field names go.

      I am sure I cannot be alone in hating the error reporting or lack of error reporting. If you have ever used Workato or similar integrations that is beautiful and it makes it super easy to determine your issues. Pipelines seems to do its best to hide what is actually happening under the hood with pointless error messages like "Something went wrong" or whatever. Even looking at the activity log it is incredibly unclear what actually went wrong most of the time.

      That isn't even getting into the actual performance issues with Pipelines. I don't see bad performance often anymore but it does happen and it is never clear why. I understand Pipelines is an outside service now but it struggles to do things in a few minutes that an Automation would have done in less than a few seconds sometimes. I am so sad they are taking away Automations and replacing it with Pipelines when it is not a 1:1 service at this point. It is not as reliable as Automations in my opinion, not by far. I wish they just kept Automations like they did with Actions when they replaced that....

      We currently only use Pipelines for things that are not business critical for the most part and that says a lot. We have moved everything we can to a service that costs us a lot in comparison but the reliability pays for itself 10x over in many cases. There are just things we are not comfortable using a Pipeline for when it matters if it works. If it can be down for a day or so then Pipleines is a good candidate for moving that data.
      • PaulPeterson1's avatar
        PaulPeterson1
        Qrew Assistant Captain
        3 years ago
        Thank you, after a couple of days of no response I was beginning to feel like I was the only one.  I am disappointed at the lack of response from Quickbase.

        ------------------------------
        Paul Peterson
        ------------------------------
  • JohnKaras1's avatar
    JohnKaras1
    Quickbase Staff
    3 years ago
    Hi Community members - Thanks for the feedback here, we appreciate the frustration you're voicing and really want to improve your experience. For those of you who I don't know yet, my name is John Karas and I'm a Group Product Manager here at Quickbase. I lead our a team of Product Managers that are driving Quickbase's integration & automation product strategy. One of the critical pieces to this is our Pipelines capabilities.

    Paul, Joel, Mike, Austin, and Mark, you've really shed some light on issues that we are eager to address for you and the rest of our community on the Pipelines front. We have a sleek new look & feel coming to the "Pipelines GUI" (we call it "Pipelines Builder", the area where you create and edit your Pipeline details) and we'll be unveiling more at Empower 2022 on this. I strongly believe this will help in your GUI performance pains / loading aspects. Another thing I felt here was a better desire for Error Handling / Messages. This is something we have on our radar, and we will be bring some benefits into the new Pipelines Builder. However, I think there is more work we can do on the error handling / messaging front as well beyond the areas we've got active development occurring on. I also felt pain here around not being able to re-order / insert / move steps. This new Pipelines Builder is designed to enable this exact functionality. The feedback on field-name changes, step & error reporting, and additional experience improvements are all valid and my team is indeed looking at how we can tackle them.

    Your feedback helps us continuously prioritize & reprioritze what things we build & deliver to make our customers jobs & lives easier. As with anything in software (or in life), we know there are bright spots and there are areas that need more attention. We hope you'll continue using this powerful platform as we continue to use your feedback to make it more delightful. I hope to see many of your Empower, and would be happy to spend some more time together catching up & hearing more of your thoughts.

    ------------------------------
    John Karas
    ------------------------------
    • MikeTamoush's avatar
      MikeTamoush
      Qrew Commander
      3 years ago
      John,

      I'm glad to hear that you are working on some of the more major issues. The 2 things I would hope you would seriously consider:

      1. Loading the list of pipelines is painfully slow. Once I transfer my automations, i will likely have 200+ pipelines. Even with the 70ish I have now, I can't load the Pipelines Dashboard in less than a couple minutes (my current solution is to filter by tags and then bookmark each tag). Every time i search for a Pipeline, and clear the search - another couple minutes just waiting for load.

      2. Please extend the deadline for when you remove automations, until the Pipeline platform is up to speed. It does not make sense to force the change before Pipelines is ready.

      ------------------------------
      Mike Tamoush
      ------------------------------
    • JeremyAnson's avatar
      JeremyAnson
      Qrew Cadet
      3 years ago
      Thanks for the update John - great to hear improvements are on the way.

      At the risk of repeating some of the other comments, here is a 'wishlist' I shared with our Customer Success Manager last year.

      1. Access to all pipelines

      While I do have the option to 'switch to another user' when viewing pipelines, it seems details in the activity log are not visible when viewing as another user, making it impossible to investigate problems. Is this a known issue? Are there any plans to make it easier to view all pipelines, regardless of who created them?


      2. Basic editing controls

      Is there a plan to add save, undo/redo to the editor or to enable view/edit modes? It's too easy to accidentally change a pipeline.

       3. Adding comments

      The pipeline editor would be much improved if we could add comments. It's impossible to understand what a pipeline is doing without accessing each step because the editor only shows the step name (On New Event, Search Records, Update Record etc.).

       4. Copying applications/creating a sandbox

      Is there any plan to replicate pipelines when an application is copied? Doing this manually adds a significant overhead to our use of pipelines. We have contemplated building our own utility to automate this process, but would rather not have to do this.

      5. Modifying pipelines

      It seems to be very difficult to move and insert steps in existing pipelines and impossible to change the trigger step. Changing pipelines by exporting the YAML, editing and importing as a new pipeline is not a great solution and we usually end up starting again from scratch.

      6. Notifications of errors

      Are there any plans to provide notifications when pipelines generate errors?

       7. Pipeline performance

      Ideally we would use pipelines to provide some back-end functionality that would integrate with the front-end – i.e. user action on screen triggers a pipeline and the results are reflected back on screen – but I appreciate this cannot be guaranteed, and that with the current performance we're implementing workarounds to prevent users being confused and frustrated that their actions take time – often several minutes - to be reflected in the application.

      I know there have been steps taken to increase the resources dedicated to processing pipelines, but is there a longer-term road map in place to improve performance of pipelines?

      8. Analysing pipeline performance

      Are there any plans to provide tools to enable us to analyse pipeline usage and performance and perhaps to prioritise some pipelines over others?



      ------------------------------
      Jeremy Anson
      ------------------------------
    • PaulPeterson1's avatar
      PaulPeterson1
      Qrew Assistant Captain
      3 years ago
      John,

      Thank you for taking the time to provide your feedback.  I look forward to the changes and hope the "new" builder resolves some of the issues.  I still remain skeptical that pipelines are truly in a position to replace automations.  There still seem to be several bugs to address in addition to the lag between trigger, pipeline execution and pipeline completion.

      One issue that I did not hear you address is the primary issue for opening this thread.  I found a way to create a pipeline to accomplish the desired result without but there are two issues.  First, I still cannot search one of the tables so the pipeline is only doing half of the work, second, the pipeline ran for just over 90 seconds.  With that kind of performance, I'm terrified of a situation where a few hundred records may trigger at one time.  IF support is able to resolve my open case regarding this issue, I will likely return to my original approach for building this pipeline.

      Like many other developers, I have a little over 100 Automations to convert into pipelines.  I can't afford to burn an entire day converting each Automation to a pipeline.  My experience with the tool to convert Automations to Pipelines has been laughable.  This tool wasn't capable of converting a basic automation that updates a field based on a trigger.  I'd be afraid to see what it would do with something more complex.

      I will try to remain optimistic and continue to crunch away at converting my automations.

      ------------------------------
      Paul Peterson
      ------------------------------
  • JoelHickok's avatar
    JoelHickok
    Qrew Cadet
    2 years ago

    Did anybody else notice that the Product Manager made a quick appearance and then dissapeared again forever?  It must be a bit embarassing given that now the new Pipeline "designer" is available in Beta, and it has already corrupted 3 of my existing Pipelines PERMANENTLY.  I'm sure one of the development team members could send my  source YAML over to re-create, but I am wary enough that I exported the YAML on my own before even touching their new tool.  So thankfully I had a backup for all 3 corruptions.

    We are expected to waste our time (and the time of our employers) being the guinea pigs on this beta tool.  It offers some much needed improvements, thankfully, but is still a bit of a mess once you really dig in.  I hate to say that for now it's still better to use the clunky "old" original Pipeline builder.