View Only
Expand all | Collapse all

Pipeline GUI Issues

  • 1.  Pipeline GUI Issues

    Posted 03-25-2022 09:00

    I am wondering if anyone else is encountering troubles creating pipelines using the GUI.  I spent most of the day yesterday attempting to create a very simple pipeline that is triggered by a date change and updates that change in another table.  I had a similar Automation that took less than 5 minutes to create.

    The overall layout of the pipeline:

    Step A - Triggered when the date changes

    Step B - change a field in the same table that will be used in the search (as I'm typing this a thought occurred, is the prev option available in the advanced query?  for {{fid.XEX.a.$prev.field_name}}?  I know the syntax is probably wrong, but will that work conceptually)

    Step C - Prepare Bulk Record Upsert - All three fields needed for the upsert are defined

    Step D - Search for changed records (even though in most cases the search will only return 1 record)

    In Do Loop:

    Step E - Add a Bulk Upsert Row 

    Step F - Update Record in Step A to clear the search flag

    Loop Complete

    Step G - Commit Upsert

    The problems I encountered were: 

    A)  Each time I something to a step (trigger field, fields for future use, table, etc.) I get a spinning circle of dots for at least a minute and cannot edit anything while waiting for the precious change to update.

    B)  Table, fields, or other required data IS entered and showing in the step but I still get the yellow caution when the step is minimized stating that the step is missing required fields.  Most of the time this will go away if I refresh the schema (IF this works, I have to refresh the Schema several times).  Other times the caution will only go away after refreshing the browser window.

    C)  The BIGGEST issue, I am not able to add the fields to the Add a Bulk Upsert Row.  They are defined in Step C - Prepare Bulk Record Upsert.  When I add the step to Add a Bulk Upsert Row I get two error messages.  The first, and most annoying one, just says "Oops, something went wrong."  That has to be the most frustrating and useless error message I have ever seen.  When I refresh the schema I get, "Internal error while refreshing schemas.  Please contact support if you need help"  This message is just as useless.  I have contacted support and after waiting an entire day the issue is no closer to resolution.

    Thank you for your patience if you made it this far into my rant.

    Is anyone else experiencing the same issues?  If so, how did you resolve them?

    @Evan Martinez - Are there any plans to improve the error message in the pipeline GUI?  My experience with them in this case and other instances has been unpleasant.  Are there any plans to make the error messages less vague and ambiguous?  I know error handling can get complicated, but what purpose does "Oops, something went wrong" serve?  Also, are there plans to improve the performance of the pipeline GUI.  I was extremely excited when pipelines were introduced, but have become much less of a fan after trying to creating them.

    Paul Peterson

  • 2.  RE: Pipeline GUI Issues

    Posted 03-28-2022 10:47
    Hi Paul,

    I feel your frustration, and cannot provide any specific solutions to your problems right now.  However, I will echo that my experience using Pipelines has been a double-edged sword.

    Pipelines have introduced many powerful features and functions for our organization.  Some of these we already feel are indispensable.  But the User Interface and Pipeline development experience is miserable.  I am shocked that the Quickbase Team has allowed this product to exist for so long now without at least resolving some of the biggest issues.  And now Automations are already on the path for deprecation, while Pipelines don't feel honed in perfectly yet.

    One of my biggest complaints is that you can not re-order steps in the UI, and you cannot insert a brand new trigger as the first step to replace the existing trigger.  You can download the YAML file, carefully edit it, and then create a new Pipeline using that.

    Joel Hickok

  • 3.  RE: Pipeline GUI Issues

    Posted 03-28-2022 15:25

    I'm just here to echo everything Joel said, in hopes QB Development takes notice.

    The user interface can be quite miserable, specifically not being able to reorder steps. Also, if you change the name of a field that you use in Jinja code, you can break your pipeline (it will continue to work unless you go to edit the pipeline in any way, at which point it will break).

    I get a number of error messages etc like you, and like Joel said, the fact they are removing automations before they really fix the issues is disappointing.

    Mike Tamoush

  • 4.  RE: Pipeline GUI Issues

    Posted 03-28-2022 16:15
    Edited by Austin K 03-28-2022 16:17
    You can re-order steps in the UI. Drag and drop but it doesn't always work exactly as you want. However your point stands about the trigger not being able to be replaced easily if at all in some cases through the UI. I only recently learned you could drag and drop in here lol.

    My biggest complaint about Pipelines is the fact that it hides so much from you and it also seems to work based on the field name and not the field ID which seems insane to me. If you have a field that has a name change the Pipeline breaks. Not only does it break but a schema refresh won't fix your issue. You need to go to the original, and possibly all, steps the field is used in and click into the box with the field list. This is what updates the step to use the new schema after it gets refreshed, at least as far as the field names go.

    I am sure I cannot be alone in hating the error reporting or lack of error reporting. If you have ever used Workato or similar integrations that is beautiful and it makes it super easy to determine your issues. Pipelines seems to do its best to hide what is actually happening under the hood with pointless error messages like "Something went wrong" or whatever. Even looking at the activity log it is incredibly unclear what actually went wrong most of the time.

    That isn't even getting into the actual performance issues with Pipelines. I don't see bad performance often anymore but it does happen and it is never clear why. I understand Pipelines is an outside service now but it struggles to do things in a few minutes that an Automation would have done in less than a few seconds sometimes. I am so sad they are taking away Automations and replacing it with Pipelines when it is not a 1:1 service at this point. It is not as reliable as Automations in my opinion, not by far. I wish they just kept Automations like they did with Actions when they replaced that....

    We currently only use Pipelines for things that are not business critical for the most part and that says a lot. We have moved everything we can to a service that costs us a lot in comparison but the reliability pays for itself 10x over in many cases. There are just things we are not comfortable using a Pipeline for when it matters if it works. If it can be down for a day or so then Pipleines is a good candidate for moving that data.

  • 5.  RE: Pipeline GUI Issues

    Posted 03-28-2022 17:08
    Thank you, after a couple of days of no response I was beginning to feel like I was the only one.  I am disappointed at the lack of response from Quickbase.

    Paul Peterson

  • 6.  RE: Pipeline GUI Issues

    Posted 03-28-2022 17:13
    They do know about error reporting and delayed the end of life for creating new Automatons until about July 1st so they can address error reporting and Governance (who can access each other's Pipelines).

    There other points you raise which are valid and we have to hope that there is ongoing plans to improve Pipelines and Documentation for the various Channels.

    Mark Shnier (YQC)

  • 7.  RE: Pipeline GUI Issues

    Posted 03-28-2022 18:46
    Yes, I've known about the drag-and-drop for awhile now, and it is so inconsistent you might as well say it's not an option.    I'm sure you will find that it does not work well, and it does not work all the time.

    Joel Hickok

  • 8.  RE: Pipeline GUI Issues

    Posted 03-29-2022 18:22
    Hi Community members - Thanks for the feedback here, we appreciate the frustration you're voicing and really want to improve your experience. For those of you who I don't know yet, my name is John Karas and I'm a Group Product Manager here at Quickbase. I lead our a team of Product Managers that are driving Quickbase's integration & automation product strategy. One of the critical pieces to this is our Pipelines capabilities.

    Paul, Joel, Mike, Austin, and Mark, you've really shed some light on issues that we are eager to address for you and the rest of our community on the Pipelines front. We have a sleek new look & feel coming to the "Pipelines GUI" (we call it "Pipelines Builder", the area where you create and edit your Pipeline details) and we'll be unveiling more at Empower 2022 on this. I strongly believe this will help in your GUI performance pains / loading aspects. Another thing I felt here was a better desire for Error Handling / Messages. This is something we have on our radar, and we will be bring some benefits into the new Pipelines Builder. However, I think there is more work we can do on the error handling / messaging front as well beyond the areas we've got active development occurring on. I also felt pain here around not being able to re-order / insert / move steps. This new Pipelines Builder is designed to enable this exact functionality. The feedback on field-name changes, step & error reporting, and additional experience improvements are all valid and my team is indeed looking at how we can tackle them.

    Your feedback helps us continuously prioritize & reprioritze what things we build & deliver to make our customers jobs & lives easier. As with anything in software (or in life), we know there are bright spots and there are areas that need more attention. We hope you'll continue using this powerful platform as we continue to use your feedback to make it more delightful. I hope to see many of your Empower, and would be happy to spend some more time together catching up & hearing more of your thoughts.

    John Karas

  • 9.  RE: Pipeline GUI Issues

    Posted 03-29-2022 18:29

    I'm glad to hear that you are working on some of the more major issues. The 2 things I would hope you would seriously consider:

    1. Loading the list of pipelines is painfully slow. Once I transfer my automations, i will likely have 200+ pipelines. Even with the 70ish I have now, I can't load the Pipelines Dashboard in less than a couple minutes (my current solution is to filter by tags and then bookmark each tag). Every time i search for a Pipeline, and clear the search - another couple minutes just waiting for load.

    2. Please extend the deadline for when you remove automations, until the Pipeline platform is up to speed. It does not make sense to force the change before Pipelines is ready.

    Mike Tamoush

  • 10.  RE: Pipeline GUI Issues

    Posted 03-30-2022 03:33
    Thanks for the update John - great to hear improvements are on the way.

    At the risk of repeating some of the other comments, here is a 'wishlist' I shared with our Customer Success Manager last year.

    1. Access to all pipelines

    While I do have the option to 'switch to another user' when viewing pipelines, it seems details in the activity log are not visible when viewing as another user, making it impossible to investigate problems. Is this a known issue? Are there any plans to make it easier to view all pipelines, regardless of who created them?

    2. Basic editing controls

    Is there a plan to add save, undo/redo to the editor or to enable view/edit modes? It's too easy to accidentally change a pipeline.

     3. Adding comments

    The pipeline editor would be much improved if we could add comments. It's impossible to understand what a pipeline is doing without accessing each step because the editor only shows the step name (On New Event, Search Records, Update Record etc.).

     4. Copying applications/creating a sandbox

    Is there any plan to replicate pipelines when an application is copied? Doing this manually adds a significant overhead to our use of pipelines. We have contemplated building our own utility to automate this process, but would rather not have to do this.

    5. Modifying pipelines

    It seems to be very difficult to move and insert steps in existing pipelines and impossible to change the trigger step. Changing pipelines by exporting the YAML, editing and importing as a new pipeline is not a great solution and we usually end up starting again from scratch.

    6. Notifications of errors

    Are there any plans to provide notifications when pipelines generate errors?

     7. Pipeline performance

    Ideally we would use pipelines to provide some back-end functionality that would integrate with the front-end – i.e. user action on screen triggers a pipeline and the results are reflected back on screen – but I appreciate this cannot be guaranteed, and that with the current performance we're implementing workarounds to prevent users being confused and frustrated that their actions take time – often several minutes - to be reflected in the application.

    I know there have been steps taken to increase the resources dedicated to processing pipelines, but is there a longer-term road map in place to improve performance of pipelines?

    8. Analysing pipeline performance

    Are there any plans to provide tools to enable us to analyse pipeline usage and performance and perhaps to prioritise some pipelines over others?

    Jeremy Anson

  • 11.  RE: Pipeline GUI Issues

    Posted 03-30-2022 08:42

    Thank you for taking the time to provide your feedback.  I look forward to the changes and hope the "new" builder resolves some of the issues.  I still remain skeptical that pipelines are truly in a position to replace automations.  There still seem to be several bugs to address in addition to the lag between trigger, pipeline execution and pipeline completion.

    One issue that I did not hear you address is the primary issue for opening this thread.  I found a way to create a pipeline to accomplish the desired result without but there are two issues.  First, I still cannot search one of the tables so the pipeline is only doing half of the work, second, the pipeline ran for just over 90 seconds.  With that kind of performance, I'm terrified of a situation where a few hundred records may trigger at one time.  IF support is able to resolve my open case regarding this issue, I will likely return to my original approach for building this pipeline.

    Like many other developers, I have a little over 100 Automations to convert into pipelines.  I can't afford to burn an entire day converting each Automation to a pipeline.  My experience with the tool to convert Automations to Pipelines has been laughable.  This tool wasn't capable of converting a basic automation that updates a field based on a trigger.  I'd be afraid to see what it would do with something more complex.

    I will try to remain optimistic and continue to crunch away at converting my automations.

    Paul Peterson

  • 12.  RE: Pipeline GUI Issues

    Posted 04-27-2022 14:32
    John, Thanks for the update. 

    This cannot come soon enough.  Every week I continue to pain and labor over what has become a dreaded part of my job, and I continually am frustrated at the QB Team for promoting Pipelines so fervently before they were truly ready for the End User to have a good experience.  It's hard to wait longer, but I hope these updates are "the cat's meow," as they say. 

    Even right right now, today, I am rebuilding a Pipeline from scratch simply because I now need it to fire on Create AND Update record, not just Create as it was setup.  Wasted time for me.  And my overall confidence is low due to the sheer number of other issues I have seen Users report, that you and other Quickbase staff have not yet openly addressed.

    Another User in this thread summed it up, when they wrote: "I still remain skeptical that pipelines are truly in a position to replace automations."  Even though they are more powerful than Automations, Pipelines are not reliable for now, and I do not believe they will be for 1-2 more years, judging by the pace things have been moving.

    Joel Hickok

  • 13.  RE: Pipeline GUI Issues

    Posted 05-17-2022 11:57
    Hi John,

    Having watched the Empower presentation on the roadmap for pipelines I'm pleased to see the planned improvements to the editor, the addition of error notifications and the ability for admins to view all pipelines. The focus on supporting pipeline growth and meeting performance expectations is also very welcome, and it's clear the pipelines team are listening to and responding to feedback.

    We've shared several additional pain points and requests through various channels back into QB and it would be great to understand if these are on the roadmap.

    1. You mentioned that teams were asking for pipelines to be copied with applications, but it wasn't clear if this was on the roadmap. We have contemplated building our own utility to semi-automate this as it's a major headache whenever we contemplate creating a copy of an application for testing, but would obviously like to avoid doing this if we don't need to.

    2. Support for UTF-8 - I've been told by the support team that we cannot switch on UTF-8 support and use pipelines with triggers, which - as we cannot work without these pipelines - means we cannot switch on UTF-8; it's very hard for our global business to understand why we're in this position. 

    3. Basic editing controls - Is there a plan to add save, undo/redo to the editor or to enable view/edit modes? It's too easy to accidentally change a pipeline.

    4. Adding comments - It's impossible to understand what a pipeline is doing without accessing each step because the editor only shows the step name (On New Event, Search Records, Update Record etc.) and being able to add comments (e.g. 'When a Project is ready to be archived', 'Find tasks related to the project', 'Update task status to archived') would be a great time saver.

    5. Analysing pipeline performance - Are there any plans to provide tools to enable us to analyse pipeline usage and performance and perhaps to prioritise some pipelines over others?

    If you have time to comment on any of these points it would be greatly appreciated.


    Jeremy Anson