Creating CSV with Pipelines/Paginating with REST API
I want to first note that I've seen the really excellent explanations by Sharon Faust (QuickBase Junkie), Scott Galloway, and what's on QB University for doing something similar (but less complex, I guess) to what I'm about to describe. I'll link to those at the end in case anyone stumbling upon this could benefit from those.
Ultimately what I'm trying to do is use the Audit API to generate a CSV file and drop it into a third party cloud storage location. Most of the individual pieces of this are in place. I can pull data from the Audit API. I can generate a CSV by pushing query results into a Bucket channel pipeline table (if it doesn't require pagination, which is the big issue – more on that in a minute). I can then use the "Download CSV" step to grab a download URL for the CSV. I can then use the "Fetch CSV" step to grab the CSV data, which I then send to the third party cloud service with a "Make Request" step. The problem for me is that the Audit Log generates tens of thousands of rows per day, so I'll need to paginate through it and build out the CSV file. I can do that with callable pipelines. The problem is that the only generation of CSV data can happen within a Pipeline Bucket channel table (not to be confused with the other table under the Bucket channel that doesn't have a CSV step – see attached picture for reference). I don't see a way to reference the Bucket channel table with each iteration of the callable pipeline so that I can continue to build out the table until it's filled with all the rows from my original Audit query. I feel like I'm missing something obvious here, but I have hit a wall. If I happen to figure out anything on my own, I'll be sure to post here. Just hoping someone else has dealt with this and can nudge me in the right direction.
For reference:
QuickBase Junkie's "How to use Pipelines to email a CSV file in Quick Base" - https://www.youtube.com/watch?v=gra6zBkeg0g
QB University's "Archive data with the Bucket channel" - https://university.quickbase.com/build-powerful-pipelines-using-included-features/647381
Scott Galloway's Empower Session "API Pagination with Callable Pipelines with Scott Galloway" - https://university.quickbase.com/empower-2021/849480
Thanks so much in advance for your time!Christine
------------------------------
Christine Suarez
------------------------------
