Pipelines - Is This A Good Use Considering API Limits?
We have some fairly large tables of data in QB and then we need to update the table based on results that we have collected into a CSV.
We have done this externally in Javascript but would love to keep all of this tied to QB instead of having another platform (AWS) etc.
Basically in Javascript, we read the entire table into a local DB, then bring in the CSV data, then for each line of the CSV data, see if we have a match already in the QB Table. If we do not, then we add that to a list and once we process the entire CSV file, then we can efficiently upload all the "need to add" records at one time and not run up API requests.
Are there efficient way to do similar in Pipelines/Jinja? I'm sure we could CSV line by line, search the DB if we have a match and upload if we don't but would be a mess from number of API calls required (I think).
Happy to dive in if sounds pretty doable just didn't want to waste a bunch of time on something that would obviously not work to people who have been there, done that.
------------------------------
Chris Anderson
------------------------------
We have done this externally in Javascript but would love to keep all of this tied to QB instead of having another platform (AWS) etc.
Basically in Javascript, we read the entire table into a local DB, then bring in the CSV data, then for each line of the CSV data, see if we have a match already in the QB Table. If we do not, then we add that to a list and once we process the entire CSV file, then we can efficiently upload all the "need to add" records at one time and not run up API requests.
Are there efficient way to do similar in Pipelines/Jinja? I'm sure we could CSV line by line, search the DB if we have a match and upload if we don't but would be a mess from number of API calls required (I think).
Happy to dive in if sounds pretty doable just didn't want to waste a bunch of time on something that would obviously not work to people who have been there, done that.
------------------------------
Chris Anderson
------------------------------
