A solution to un-pivot table data

If this topic has been discussed and there are well known solutions to this topic - I apologize. I didn't come across any before I came up with the solution below.

Background:

I was working on a request from my team to build out some functionality that required a lot of check boxes.  I didn't want to take a lot of time building a custom form to ensure that the checkbox data was stored with a row for each box. There are 28 checkboxes representing various issues (oil leak, lights, forklift forks, etc). While not totally necessary, I did hope I could somehow un-pivot these columns into a table where only issues that were checked had a record.

TLDR;

Using fetch JSON and the Quickbase API, you're creating a list of field ids that you want to unpivot and then for each record in a second set of JSON data, you are looping through that field list and creating a new single record in a 2nd table with the values you are unpivoting as well as any other data you wish to repeat in each new row.  This process leaves the original table completely intact.

Solution:

API Urls

1. You'll need two API request urls. 
   1. One request for the field information for the table that has the information you wish to un-pivot (eg, field id, field label, etc)
   2. One request to a report that has the data you'll need for the records in the new table. (eg. values in fields, checkbox data, etc) Don't forget to include columns that contain foreign key data you'll use in the new table
2. If you know how to create/build those URLs, skip ahead to step 9
3. To build/learn about how to build the API URLs, go to developer.quickbase.com 
4. On the left side of your window you'll see a list of different topics related to the Quickbase API. Select the arrow next to the 'Fields' label.
5. Then, click on the first item on the list "Get fields for a table"
6. From here, you can enter the table id of the table you want to un-pivot, and fill out the other information related to realm, authorization, etc. I suggest also testing it and seeing the data that comes through.
7. Copy the url produced in the top right corner.  This is your field data URL.
8. Back on the list on the left, click on 'Run a report', and then repeat steps 6-7 to get the record data URL.

Pipeline | Fields Data

1. Create a new pipeline and add a 'Fetch JSON' step.
2. Populate the fetch JSON  step with the field data url and required headers. This step is a GET step.
3. Following that step, create a 'Iterate over JSON' step
4. The step should automatically select the prior step, but in case it didn't, in the 'JSON Source' field, select the previous step as the source.
5. In this step, it is helpful to have a sample of the JSON schema so you can reference some of the items in the next step. So include a sample data dump from the developer.quickbase.com API website. This makes references to specific field data much simpler
6. In the 'Iterate over JSON' step, go to the bottom and filter the field list to ONLY include the fields you want to unpivot.

In my case this was simple as all of my fields were checkboxes, so I just chose the 'Field Type' field as my filter, and conditioned it on "checkbox". 

If you aren't as fortunate as me, and you don't use the 'Field Help' field for your applications - you could use the 'Field Help' field as your filter by populating each field that you wish to un-pivot with the text 'unpivot', and then filter on the 'Field Help'

This is absolutely vital to the process. If you can't get the field list down to exactly what you want you'll get unwanted rows.

Pipeline | Record Data

1. The 'Iterate over JSON' step from above will have created a 'loop' step. IN BETWEEEN the 'Iterate over JSON' step and its corresponding loop,  repeat the steps 1 - 4 from the Pipeline | Fields Data section for the Record Data url
2. A second loop will have been created as part of this new 'Iterate over JSON' step
3. Move the Field Data Loop (from the 1st part) INTO the new loop you just created. 
   1. The field data, i.e. the information that has field ids, field labels, etc. should be nested INSIDE the loop that has the table data, i.e. the records that have the actual information you want to un-pivot.

Testing for Field Data

1. In the nested loop (the field data loop), create a 'condition' step.
2. In the drop down field where you select the field to evaluate, go to the bottom of the list and choose "Expression (advanced)".
3. In the dropdown to the right, you can leave it as "evaluates to True"
4. In the criteria field below the two dropdowns, include the following jinja

{{d.raw_record[(b.id|string)]['value']}}

• To explain, what we're doing here is taking the table data we downloaded (d.raw_record), and using the current field id (b.id) of our loop to grab the data ['value'] for just that specific field. In my case, the value was either going to be 'true' or 'false' because it was a checkbox. You may need to adjust the logic to test if the field has data or not. If you don't test for this, you'll get empty rows of data.

Creating a Record

1. If you haven't already, create the table where the data is going to go. 
2. In the pipeline, in the 'If condition is met' branch, add a 'Create Record' step
3. Choose the table you created
4. Add all the fields from that new table that you are going to populate with data from the un-pivot.
5. For each of the fields in the NEW table that you'll use to group on, use jinja to select the fields that will fill each of the new fields.

I've included the jinja I used to populate the fields that get the same data for every record.  This was the record id of the report that has all the checkbox issues

{{(d.raw_record['3']['value'])|int}}

Below was the date of the report which I wanted to include in each new record

{{time.parse(d.raw_record['6']['value'])|date_mdy}}

For the field that contains the value that you want to un-pivot, you use the same jinja statement you used in the condition section to check on valid data.

{{d.raw_record[(b.id|string)]['value']}}

Cross your fingers, say a few prayers, call your mother and then hit run on the pipeline.