Skip to content

6. Analyzing data

Jump to bottom
Lauren Fink edited this page Mar 25, 2022 · 2 revisions

Analyzing data

Opening and reading binary .gdf files

In Terminal, launch Julia. If you don't have Julia or don't know where it's located, this link will help.

# Change directory to the location of GEMDataReader.jl (in the GUI sub-dir)
# Remember that ; switches you between Julia and shell
# Then do the following:
!("." in LOAD_PATH) && push!(LOAD_PATH, ".")
using GEMDataReader

# Example to convert 1 file
# Convert the file "GEM_test1.gdf" to .csv in the same directory with the same filename
ifile = "/Users/laurenfink/Documents/Arduino/20180109/GEM_test1.gdf"
ofile = splitext(ifile)[1] * ".csv"
convert_file(ifile, ofile)
# Note that a .json file containing all experiment meta-data will also be output in the same directory

# Example to convert a directory of files
# This function is likely more useful and what you will want to use after you have a directory of data. The function will search subdirectories as well. 
ofiles = convert_files("../Data/")
# For each .gdf file, there will now be two output files with the same file name but with extensions .csv and .json.

Example analyses provided in the GEM POC repository:

In the repository linked above we provide final data tables and code to recreate the figures and statistical analyses reported in our initial papers. All analyses are written in Python, using Jupyter Notebook.
If you don't have Jupyter, here are some installation instructions.

Once you have Jupyter, you can analyze your own tapping experiment data with our analysis methods. Simply:

  • navigate to your terminal or shell.
  • change directory to the location of our analysis files.
  • type "jupyter notebook" to launch the Jupyter Notebook App, which will appear in a new browser window.
  • if you want to run our analyses, specify all file paths correctly at the top of the provided Jupyter notebook

Testing data throughput

The time it takes to transfer the memory into the right buffer @ 100ms delay between packets

  • packet size of 10 bytes = < 1ms delay (getting zeros so can't measure)
  • packet size of 17 bytes (the size of GEM packets) = < 1 ms per packet; avg. 113 bytes / ms (getting zeros so can't measure)
  • packet size of 255 bytes = avg 16.55 ms (15.4 bytes / ms)

We can assume that the aArduino can achieve the baud rate.. but what about python side? 14.2 kb/sec

  • writing and reading from our python buffer

Can python keep up at 1ms delay, 255 bytes? Yes! Bandwidth of our total system is more than sufficient Testing to make sure the system can handle a much bigger data stream than what we're actually asking it to handle in our experiments.

  • driver, pyserial, our code (which just funnels data into file)
  • avg of 12 bytes / ms is our transfer rate (14 bytes/ ms is about max given our baud rate)
  • 20.9 ms is the avg time it takes to write packet
  • all data there!
  • 444ms to send 20 packets of 255 bytes Get exactly the same avg when sending data as an array of 255 bytes vs. sending individual bytes.
  • we are only sending an array of 4 int 16s.. so again well within our range of capability. And python getting all data no problem.
Clone this wiki locally