Time Series Data

  • 1
  • 1
  • Question
  • Updated 2 years ago
  • In Progress
I have a large amount of time series data taken in 30 second increments over the course of 4-5 days.  This gives 10,000 individual items needed to store in 12+ columns.  I do not want to analyze the data in QB, due to the 200 item limit, but I do want it stored together.  I imported the CSV data and it is held just fine, but when I go to add it to a project description table I made, then I have to enter the data in individual increments.

Is there an option for a mass import, basically treating each column of 10,000 pieces of data as 1 piece?  I just need to avoid individually entering 10,000 items needlessly. 

Thanks!
Photo of Neil Shepard

Neil Shepard

  • 1,248 Points 1k badge 2x thumb
  • happily fed

Posted 2 years ago

  • 1
  • 1
Photo of QuickBaseCoach App Dev./Training

QuickBaseCoach App Dev./Training, Champion

  • 52,786 Points 50k badge 2x thumb
I am not quite sure of your table structure, but if you make a summary report one of the options under the more button is to copy these records to another table.
Photo of Neil Shepard

Neil Shepard

  • 1,248 Points 1k badge 2x thumb
See the images below for what I am talking about.  I looked at the summary, but it just said I had 10K data points.  The bottom pic show the input for a new time series and at the bottom is the selection for the project the data belongs to, a 1 to Many table.

Maybe that helps explain a bit.  I know QB is not really meant for this, but trying to find the limit of a QB in relation to a new project we are doing.

Photo of Ⲇanom the ultimate (Dan Diebolt)

Ⲇanom the ultimate (Dan Diebolt), Champion

  • 27,694 Points 20k badge 2x thumb
>Is there an option for a mass import, basically treating each column of 10,000 pieces of data as 1 piece

The option you are looking for is called a file attachment field. Store all your time series data in a file and upload the file.  I don't understand the goal of what you are trying to accomplish.

If you need some type of custom import, manipulation, calculation or access to the data after bulk import this can all be done with script.
Photo of Neil Shepard

Neil Shepard

  • 1,248 Points 1k badge 2x thumb
Long story short, we have a new project and it involves all this data.  My boss would like to see if QB can act as the repository for the data since we need to email and report on results to the client.  Having originally spec'd QB for project management, this is a new world.

We know it won't do the data analysis on such a large heap of data, not the correct tool.  For that we are looking at Python scripting with Bokeh or Pandas to do what we want for the data plotting and manipulation.  I have already created a storage table for that.

The problem lies in my having no clue how to deal with such a huge chunk of data. 
Photo of Matthew Neil

Matthew Neil

  • 31,478 Points 20k badge 2x thumb
Neil,  How you been?  QuickBase will be able to do this for you.  

To assign all those Series data records to the project, you can do a mass update, by using the "More" drop-down, then select "search & Replace in this report"

You can then select "Related Project", change 'blank' to 'your project'

Click replace.




This will update all the child records at once.
Photo of Neil Shepard

Neil Shepard

  • 1,248 Points 1k badge 2x thumb
I'm cruising along.  Things are starting to come together.

I will work on the items both you and Danimal mentioned.
Photo of Ⲇanom the ultimate (Dan Diebolt)

Ⲇanom the ultimate (Dan Diebolt), Champion

  • 27,694 Points 20k badge 2x thumb
Your project can be implemented entirely in QuickBase. D3.js is better than Bokeh and jsData (and other JavaScript libraries) can probably do everything you are doing in Pandas. See:

D3
https://d3js.org/

jsData
http://learnjsdata.com/

Bokeh
http://bokeh.pydata.org/en/latest/

Pandas
http://pandas.pydata.org/

I am an electrical engineer and I have extensive experience with modeling, simulation and data analysis - and I have written libraries to model motors (synchronous, asynchronous, etc), switchgear, semiconductors (http://ws680.nist.gov/publication/get_pdf.cfm?pub_id=19115) etc.

Again, your project can be done entirely within QuickBase.
(Edited)
Photo of Neil Shepard

Neil Shepard

  • 1,248 Points 1k badge 2x thumb
Ah so, now we should talk!!!!  We are an electrical consulting firm.  If QB can do what we want, then we would be golden.  

As Matthew Neil knows, I am not a programmer, hence the work with QB.  Python is done by a coworker who is a wizard at Python.

I will look at D3 tonight, and look through your IEEE article also. (I am IEEE PES myself).  We are doing event analysis on the time series data, graphing load to power factor, etc. to analyze what is happening electrically.  Our client started by storing and analyzing these data points in Excel and it is not pretty.  I am trying to see if QB will do this.

Will D3js import the data from QB, analyze and plot as we want, and then export the results back to QB?  If so, Would you be willing to talk some off-line so I can pick your brain on methodology for using within QB?

Thanks for the help you have given so far on this question and the others you have answered!

Cheers. 
Photo of Ⲇanom the ultimate (Dan Diebolt)

Ⲇanom the ultimate (Dan Diebolt), Champion

  • 27,694 Points 20k badge 2x thumb
D3 can be used within QuickBae as your charting package drawing data out of CSV file attachments as needed. D3 has an excellent CSV parser which can part CSV to an array of objects and back to CSV for storage back to a file attachment.

Best time to speak would be Friday as I am travelling this week.