Decide on database type: Is sqlite sufficient?

[vincentvanhees]

No description provided.

[vincentvanhees]

Possible requirements:

• Ability to put the data set on a server and allow for multi-user interaction.
• Data size may grow, so scaling is a concern.
• Considering that not all data can be fully open access, some means of controlling data access per user is required.
• Anticipated data characteristics:
  • 5 million rows: 100 (participants) x 5400 seconds per person x 10 value per person (aggregated from real raw data)
  • 70 columns, most likely split across multiple tables (30 from video, 30 from audio, 10 text related)
  • Data types: mostly time series in double, although some channels may be binary / boolean.

[vincentvanhees]

Conclusion for now after talking to Carlos and Jisk: Start with SQLite for prototyping and switch to something else (e.g. postgresql) later on.

[vincentvanhees]

• Data characteristics:
  o Probably less than 100 participants
  o 50 GB per person raw data:
   Video .mp4
   Audio .wav
   Labels and descriptions xml / txt
   Text derived from audio ??
  o Student says 100 GB derived data?
  o We have some software to extract informative data features:
   1 GB per person derived data in csv files.
   Possibly aggregate to lower resolution.
• Aims:
  o Search data based on labels or data characteristics.
  o Visualize query results
  o Train/apply machine learning to query results.
  o Export aggregated values for use outside DB
  o Share ‘data’ with community to facilite re-use
• Plan:
  o SQLite for now -> need to visualize schema
  o Simply pilot in jupyter notebook, no Django framework
  o Data sharing via surfdrive for now, but not ideal => other options?
  o How to go back to video files? Can we create specific links in DB to specific fragments of video files?
   How to host this?
   Are there similar projects?