Pipeline Help

Question

I am working on connecting MethodCRM to Quickbase so that I can pull field data from a MethodCRM form and replicate it in a Quickbase form. I have attached images showing my current pipeline setup. The challenge is building the URL for retrieving records. Right now, the URL will pull the first 100 records from MethodCRM, but I have to manually change the URL to get the next batch of records. Example: rest.method.me/api/v1/tables/Contacts?skip=10&top=10

This works, but after pulling the first 10 records, I want the pipeline to automatically update so the next run changes top=10 to top=20 (and so on), allowing the pipeline to continue pulling subsequent batches without manual edits.

Best answer by hueyal

I'm not sure if I am totally following the flow; however, in your callable pipeline, can you add skipValue and topValue arguments to the call definition? Then in the URL, use those argument values:

rest.method.me/api/v1/tables/Contacts?skip={{a.skipValue}}&top={{a.topValue}}

You would need to keep track of where you are in the process to pass those values to the called pipeline.

hueyal · Accepted Answer

I'm not sure if I am totally following the flow; however, in your callable pipeline, can you add skipValue and topValue arguments to the call definition? Then in the URL, use those argument values:

rest.method.me/api/v1/tables/Contacts?skip={{a.skipValue}}&top={{a.topValue}}

You would need to keep track of where you are in the process to pass those values to the called pipeline.

Mez · Answer

How is your pipeline called, and what are the inputs, if any?

I looked at the documentation real quick. What you could do is if there is a "nextLink" value in the response, then pass this to this pipeline for processing. You may have to use jinja to extract or you could blindly trust the url and just use it.

Sign up

Login to the community