Questions on data import
Hello Quickbase Community, I'm looking for some experienced opinions on how to approach a current requirement. I basically need to upsert 25.000 data rows each day. I have little to no control ...
Forum Discussion
Martin,
Is the output format consistent? I have got a client that gets data from 50 or so suppliers and everyone of them formats it differently AND will change the format without and warning. The last part makes normalizing the data particularly challenging.
We built a Normalization app where every Supplier has their own table so there are unique formulas to get the data the way it is needed. Included is some robust error checking to look for surprises each month.
In hindsight I wish we had used a full ETL tool like Mule.
https://www.mulesoft.com/
The process would have been slower but the tools are much more sophisticated.
------------------------------
Don Larson
------------------------------
Is the output format consistent? I have got a client that gets data from 50 or so suppliers and everyone of them formats it differently AND will change the format without and warning. The last part makes normalizing the data particularly challenging.
We built a Normalization app where every Supplier has their own table so there are unique formulas to get the data the way it is needed. Included is some robust error checking to look for surprises each month.
In hindsight I wish we had used a full ETL tool like Mule.
https://www.mulesoft.com/
The process would have been slower but the tools are much more sophisticated.
------------------------------
Don Larson
------------------------------
Good point Don. I cannot be sure the output will remain constant with multiple data sources attached in the future.
Also data quality can be an issue so error checking will be important in the process. Thanks for your input.
------------------------------
Martin Suske
------------------------------
Also data quality can be an issue so error checking will be important in the process. Thanks for your input.
------------------------------
Martin Suske
------------------------------