Automations to Pipelines: Migration Project Management
With the recent announcement of our plan to retire Quickbase Automations in January 2023 many Admins have begun the process of planning how they will migrate their automated workflows to Pipelines and training their admins to take advantage of the new functionalities and workflows. To help users begin to migrate we launched our Automation Migration Tool which helps admins to transform some of their most commonly used Automations into Pipelines. Knowing that a migration task can have a lot of moving parts beyond just that transferring of Automations to Pipelines for admins, this article is going to highlight key points to consider when planning the migration of Automations to Pipelines, as well as two different approaches to the project.
To help with this migration Bree Mackey, a Solutions Architect here at Quickbase, created the Automation Migration Manager application that can now be found in our app exchange. You can find the Automation Migration Manager app in our Exchange by heading to your My Apps page in Quickbase and selecting the explore sample apps. This will take you to our Exchange of example applications and if you search Automation Migrations you will find Bree's app ready to be added to your apps. This article assumes you will make use of the Automation Migration Manager application available in the App Exchange (how-to instructions for the application are included below!), or have a different, in-house system to track your project.
Table of Contents
- Basic Considerations
- Management Styles
- Recommended Stakeholders
- Recommended Methods
- Getting started with the Automation Migration Manager App
Basic Considerations -
- How many automations do we have?
- Who manages them?
- Are they all still relevant?
- Is the process still needed?
- What resources are available to help with the conversion(s)?
- Review the Recommended Stakeholders outlined below as a guide for who should be involved.
- Assign resources at the Application, Table, or Automation level in the Automation Migration Manager app.
- What is the disaster-recovery plan?
- Define manual workarounds (where possible) and distribute to management.
- Timing for re-enablement of Automations if Pipelines need to be enhanced.
- Select target date(s) for pipelines go-live.
- Define notification process.
- It’s suggested you communicate early and often with affected users and their management.
- Define and distribute process for issue reporting if there currently is not one.
- It’s recommended to choose an off-peak or off-hours time to disable automations and enable their twin pipeline to mitigate business interruptions.
If your Quickbase realm is under central management of one person or department, there are a couple of options for approaching the conversion. Regardless of if you choose to go Application by Application or Automation by Impact, we recommend the basic considerations above be addressed.
If your realm is under distributed management, and you have a Project Management resource, it’s a great idea to leverage their experience to manage this process for you. Lacking a PM, choosing a seasoned Quickbase builder (preferably a Realm Administrator) to lead the charge will keep your teams focused and on track. It’s assumed that individual Admins will be able to access the Automation Migration Manager application to pull in their table information.
The roles listed out below are those we recommend you gather for your Project team. You may be able to have separate people for each role, or multiple roles filled by one person.
- Migration Team Lead: This role coordinates the schedules and expected resource utilization for the various roles.
- Application/Automation Coordinator: Depending on which methodology is used, you’ll want either the administrator for the application, or the Automation owner involved to complete the migration(s).
- Business Process Liaison: This person will work as the advocate for the stakeholders of a given Business Process to ensure that details regarding business-critical processes are not adversely affected.
- QA Manager: This role will develop a test methodology used to validate the new Pipelines. It is recommended to have frequent test cycles vs. one large test at the end of the migration. You should have a well-documented & unambiguous set of documents describing the success criteria for each test.
Application by Application –
This method is done by ranking all applications from most business-critical to least business-critical.
This method is considered more effective for distributed management ecosystems.
Automation by Impact –
Unlike the Application-by-Application method, this method instead focuses on the individual automations by their business criticality.
This method is considered more effective for centrally managed ecosystems.
For either methodology defined above, you can then choose to either convert the most critically ranked first, which would give additional time to re-enable automations while pipelines are dialed in, in case potential problems arise. Oppositely, beginning with the least critically ranked allows you and your resources to really build your pipeline muscles, learning optimization tricks to apply as you progress. The Automation Migration Manager application will assist you by helping keep track of which Automations have been converted, and those that have not. Since it is your application, you can add fields or tables to track any other data you find necessary!
Getting Started with the Automation Migration Manager App
Welcome to your new Automation Migration Manager application!
This article covers all the setup needed to get you up and running, and assumes the following:
- You understand the difference between an Application Manager and an Application Administrator.
- You are a Realm Administrator, have access to Realm Administrator credentials for your realm, or can have a Realm Administrator complete an action in this application.
- Quickbase UserToken
- You have permissions to create applications within the realm you will be pulling data from.
- You have a basic understanding of Quickbase Building.
1. Why Are We Doing This?
While this application will not actually DO any of the conversion work for you, we believed it was important to arm you with a tool that would help you manage this project.
Until now, there was not a way to get information about all of the automations in your realm, you would have had to go into every application individually and manually record the automations. This application will allow you to get an understanding about the scope of your project (do we have 50 automations, or 50,000?), assign resources to be responsible at the app level, table level, or individual automation level, and track conversion progress along the way.
2. Get your Realm's Application List
- Navigate to your Admin Console
- Click on Apps
- Click 'Export CSV'
- Keep that CSV handy!
3. Import Application List
- Import the Application CSV list from step 2.3 to the Application table in your Automation Migration Management application using the 'Import/Export' link on the table dashboard.
- Review the field mapping - You may have to set 'Date Created' to import to existing field. See images below.
- The native Date Created field (fid 1) has been renamed to 'Record Date Created'.
3. Pipeline Setup to Get Automations
For this step you'll need to have a Slug handy from your Pipelines dashboard. If you already have one, skip to step 3.1.
- Navigate to your Pipelines Dashboard.
- Click on your name on the upper right and select 'My Preferences'
- Under Accounts you'll find a list of all connected accounts.
- The UserToken should belong to a Realm Administrator and be assigned to the Automation Migration Manager application.
- Copy the Account Slug.
4.1 Setup Pipeline
- Navigate to the Automation Migration Management homepage.
- Click the 'Setup Pipeline' button in the left column.
- You will be taken to a new record in the Setup table.
4.2 Generate YAML
- Enter the Account Slug in the text box.
- Click outside the field on any whitespace on the form to generate the YAML.
- Select all the generated YAML and copy.
4.3 Import Pipeline YAML
- **Note these instructions are using the New Style dashboard.
- Navigate to your My Pipelines dashboard.
- Click the 'v' next to 'Create Pipeline' and select 'Import Pipeline'.
- Paste your YAML into the black box.
- The Name and Description should auto-populate. Feel free to update.
- Make sure you switch the pipeline to 'On'
5. Populate Tables Table**
That's fun to say...take a break and say it 5 times fast!
**This step is optional, however it may help you better organize your conversion process by being able to populate table names and other information along with being able to assign specific Tables to specific Resources. If you do not wish to have the table data, skip ahead to Step 5.
- Navigate to the Automation Migration Management homepage.
- Click the 'GetTable Data' button in the left column.
- When you press the button, it will only pull table data for applications where you are the Application Manager.
- If you have one user account (separate from the one you're logged in with) that owns all your realm's applications, you (or that user) must log in with those credentials to perform this step.
- If you have distributed management, each Manager will need to access this application individually and perform this step to retrieve table data from their applications.
- Your tables should automatically link up with their parent Application via existing relationship.
6. Get your Automations
Now it's time to get your automations!
- Navigate to the Automation Migration Management homepage.
- Click the 'Get Automations' button.
- Hit 'Save & Close' on the generated record.
- **If you have a large number (1,000's) of Automations, you may consider kicking off the process before you sign off for the day.
- We suggest periodically running this step to update the Automation information to keep track of your progress.
- The Pipeline will 'upsert' Automation information, meaning if the Automation ID already exists in this application, it will update that record. If it does not exist, it will create a new record.
7. Other Tables in the Automation Migration Manager Application
To enhance your project management experience converting your Automations to Pipelines, we've included some other tables we think will be useful.
- If desired, you can assign Resources to Automations or Tables to indicate responsibility for the migration.
- You will need to populate your Resources table accordingly.
- This can assist your users in understanding their responsibilities when it comes to this project.
- If you will not be assigning Resources, you may remove or ignore the relationships, or the entire table if you prefer.
- You may be in a situation where you need to divide up your resources to their respective departments.
- This table can help you understand what departments may need to borrow resources from another department in order to complete their conversions.
- You will need to populate this table accordingly.
- If you will not be using the table, you may remove or ignore the relationships, or even the entire table if you prefer.
If you're anything like us, you LOVE documentation!! No? Well, maybe you will start? If you'd like to start a data dictionary of your Pipelines, you can start tracking them in this table. It can help you understand which Pipelines replaced which Automations and who is responsible for them.
In the new style My Pipelines dashboard, click on any pipeline to get the pertinent information like name, ID, description, etc. from the right-hand pane that appears.
Pretty self-explanatory! Feel free to store documentation generated by your team in this table.
Our Automation Migration Tool is amazing, isn't it? There may be a few things you were not aware of.
- If you edit/save the Automation AFTER running the Migration Tool, we wipe out the migration information, but not the pipeline that was generated.
- You may not know that one of your admins is migrating automation multiple times. This table aims to capture the Migration information and store individual migrations in their own table.
- The Get Automations functionality can only pull the most recent migration data.
- This table is hidden from the table bar for all roles.
This table is only used to generate the YAML for the pipeline. It is hidden from the table bar for all roles.
7.7 Get Automations
This table is used to trigger the Pipeline to get your automation information. It can help you keep track of how often you've retrieved the information. It is hidden from the table bar for all roles.