Code written for my summer of 2022 internship at UT Austin (6/22/22 - 7/27/22) with Ph.D student Yefu Chen.
How to use the Data Manager app
Initial task. Take one day's (08/01/2020) OD data .csv file and process it into a given format. Used OpenCSV library. (Note: neither the data or the output are tracked in this repo because their filesizes are too large.)
Step 1: Decompress data on travel (OD) and stay-at-home dwell time collected every day from 2019-2020 to csv files, and restructure into monthly folders rather than daily for accessibility. Used Box SDK Java to access Box and OAuth 2.0 to authorize Box API.
Step 2: Process device_count and destination_count data for the first week of each month into two file, split into weekend and weekdays.
Step 3: Create basic lightweight GUI for more user-friendly interface. App created with Java Swing and can be found in the latest release. Several screenshots attached below.
Step 4: Add FileAggregator functionality to app: detect processed files and aggregate each origin's information into one row. See below for example:
| Data Status | Example | Includes |
|---|---|---|
| Raw data (~400 MB/day) |  |
|
| Processed data (~300 MB/day) |  |
|
| Aggregated data (~37 MB always) |  |
|
origin_count represents the number of devices that left the Census Block Group (CBG) in that time period, and destination_count represents the number of devices that arrived in the CBG. device_count represents the total number of devices registered to the CBG.
To get the app, find and download the latest release.
- The Box account you login with must have access to the folder "daily-social-distancing-v2".
- The folders, including "daily-social-distancing-v2" and all subfolders, must not be renamed.
- The File Processor's existing output filenames must not be changed, but you can add to the end for clarification (e.g. "weekend, weekdays, fullweek")
- The File Aggregator must not try to aggregate a month folder that already has aggregated files in it.
If you meet all of these conditions and the app still somehow breaks or doesn't work, please report an issue.
The file organizer button's function was to unzip each .gz archive in the dataset, but as that is already done, this button should not be used anymore.
To process a range of files, input the specified date range into the dialoggue box that pops up. Next, you should be redirected to a Box Authorization page. Log in to Box and click Grant Access to Box, which should redirect you to google.com. Copy the authcode from the URL (https://www.google.com/?code=AUTHCODEISHERE) and paste it into the next dialogue box. You only need to be authorized once each time you open the app.
Once you have provided all the input, the app will begin. The process should take no more than ~7 minutes per file on average. Once all the files have been processed the app will write the output file to your Desktop folder in the format YEAR_STARTDATE_ENDDATE.csv. This file should then be uploaded to the appropriate Box month folder for aggreagtion.
The data aggregator aggregates one month at a time, so input the month you want to aggregate. Then, follow the steps detailed above if the authorization dialogue pops up. Once aggregation is complete, the app will upload the output files to the appropriate Box month folder and no further action is required from you.
Are you experiencing issues when using the app? (e.g. app unresponsiveness or File Processor/Data Aggregator not doing anything) Please submit an issue and I will try to fix it as soon as possible. Include all information about which function you were using at the time, how the app is not working, etc. Thanks!
This project is licensed under the terms of the MIT License.