Forum Discussion

MartinSuske1's avatar
MartinSuske1
Qrew Member
4 years ago

Questions on data import

Hello Quickbase Community, I'm looking for some experienced opinions on how to approach a current requirement. I basically need to upsert 25.000 data rows each day. I have little to no control ...
Data import and export
MartinSuske1's avatar
MartinSuske1
Qrew Member
4 years ago
Hi Mark,

thanks for your answer. The T2T method sounds interesting and would solve the field mapping and transforming issues.

I wonder though what do you suggest for the initial import? I assume I'll have to deal with CSV files separated by semicolon, tab or comma. And I need to fill that temporary table in the first place with 25.000 rows. Which interface would provide the performance and flexibility in file structure?

I should have mentioned: In the end the whole import process should run automatically or based on some trigger. 

Best regards,
Martin

------------------------------
Martin Suske
------------------------------
MarkShnier__You's avatar
MarkShnier__You
Icon for Qrew Legend rankQrew Legend
4 years ago
For fully automated you will want the scratch file to be a connected Sync Table.  Your automated process would put the file there each day into an SFTP path or else in the cloud to Dropbox, Box, or Google Drive.

Then on a scheduled basis a Pipeline can run to do the T2T Copy and delete the scratch file records.

------------------------------
Mark Shnier (YQC)
mark.shnier@gmail.com
------------------------------