Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

124 summary csv files #175

Merged
merged 2 commits into from
Jun 27, 2023
Merged

124 summary csv files #175

merged 2 commits into from
Jun 27, 2023

Conversation

feydan
Copy link
Member

@feydan feydan commented Jun 27, 2023

Changes

  • Adds utility functions for generating CSV data from JSON including parsing JSON Schemas to get CSV headers
  • Adds 3 csv summary files to download
    • summary_totals.csv
    • summary_detected.csv
    • summary_gps.csv

Purpose

#124 describes feedback that summary files should be in csv format. This adds csv summary files.

Approach

In order to generate the csv files from JSON, some more generic functionality was needed since there is no straightforward deterministic way to convert structured nested data to a flat format.

Generally, two generic functions are added

  • getCsvHeadersFromJsonSchema - well formed csv files have a header row. It is non-trivial to generate this from json format because it may require traversing the whole object to understand which fields are needed. Additionally, some field keys may be missing. Since we have JSON schema files available to us, it is a more complete representation of all of the needed fields. This function accepts json schema and returns a flat array of headers.
  • objectToCsv - This is a generic function that can generate a csv string that can then be saved to a file. It accepts a headers array, a function that converts an object into rows of columns, and a join string. This function will ensure that all rows have the same number of columns as the headers array as a safety check.

We then use these generic functions in images.ts to generate the three above referenced files.

Pre-Testing TODOs

  • Bring the application up locally
  • Upload some files or show samples
  • Download the files and inspect the three referenced files above

Testing

This PR also contains unit tests that can be run in the frontend directory using npm run test-ts schema-parser

Copy link
Collaborator

@shollingsworth shollingsworth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@shollingsworth shollingsworth merged commit ca1a46b into aws/trashai-staging Jun 27, 2023
@shollingsworth shollingsworth deleted the 124-json-to-csv branch June 27, 2023 00:33
shollingsworth added a commit that referenced this pull request Jun 27, 2023
* Adding a link to Running-man-01/trashai_nbs project

* Bump vm2 from 3.9.17 to 3.9.19 in /infra (#162)

Bumps [vm2](https://github.com/patriksimek/vm2) from 3.9.17 to 3.9.19.
- [Release notes](https://github.com/patriksimek/vm2/releases)
- [Changelog](https://github.com/patriksimek/vm2/blob/master/CHANGELOG.md)
- [Commits](patriksimek/vm2@3.9.17...3.9.19)

---
updated-dependencies:
- dependency-name: vm2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump vite from 3.2.5 to 3.2.7 in /infra (#165)

Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 3.2.5 to 3.2.7.
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/v3.2.7/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/v3.2.7/packages/vite)

---
updated-dependencies:
- dependency-name: vite
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump vite from 3.0.4 to 3.2.7 in /frontend (#164)

Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 3.0.4 to 3.2.7.
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/v3.2.7/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/v3.2.7/packages/vite)

---
updated-dependencies:
- dependency-name: vite
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fixing build and fixing local (#170)

* Add testing instructions and auto tests (#171)

* Adding a link to Running-man-01/trashai_nbs project (#168)

* update frontend app doc

* Add manual instructions.

* Added test-ids

* Add test-ids to the navigation tabs

* Add smoke test draft

* Add placeholder file

* Add photos for manual testing instructions

* Upgrade node version

* Remove cypress

* Add test ids to summary page

* Jest + Puppeteer setup

* Add smoke test

* Added back manual instructions

* Add images for manual instructions

* Add instruction for running auto tests

---------

Co-authored-by: Dan Fey <[email protected]>

* Adding schema files to download (#173)

* adding schema files to the zip download

* revert yarn change

* fixing list

* indentation

* 124 summary csv files (#175)

* aAdding summary csv reports

* adding comments

* fixing off by 1 error in summary counts (#177)

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Dan Fey <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Zesky665 <[email protected]>
feydan added a commit that referenced this pull request Jul 1, 2023
* Adding a link to Running-man-01/trashai_nbs project

* Bump vm2 from 3.9.17 to 3.9.19 in /infra (#162)

Bumps [vm2](https://github.com/patriksimek/vm2) from 3.9.17 to 3.9.19.
- [Release notes](https://github.com/patriksimek/vm2/releases)
- [Changelog](https://github.com/patriksimek/vm2/blob/master/CHANGELOG.md)
- [Commits](patriksimek/vm2@3.9.17...3.9.19)

---
updated-dependencies:
- dependency-name: vm2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump vite from 3.2.5 to 3.2.7 in /infra (#165)

Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 3.2.5 to 3.2.7.
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/v3.2.7/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/v3.2.7/packages/vite)

---
updated-dependencies:
- dependency-name: vite
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump vite from 3.0.4 to 3.2.7 in /frontend (#164)

Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 3.0.4 to 3.2.7.
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/v3.2.7/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/v3.2.7/packages/vite)

---
updated-dependencies:
- dependency-name: vite
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fixing build and fixing local (#170)

* Add testing instructions and auto tests (#171)

* Adding a link to Running-man-01/trashai_nbs project (#168)

* update frontend app doc

* Add manual instructions.

* Added test-ids

* Add test-ids to the navigation tabs

* Add smoke test draft

* Add placeholder file

* Add photos for manual testing instructions

* Upgrade node version

* Remove cypress

* Add test ids to summary page

* Jest + Puppeteer setup

* Add smoke test

* Added back manual instructions

* Add images for manual instructions

* Add instruction for running auto tests

---------

Co-authored-by: Dan Fey <[email protected]>

* Adding schema files to download (#173)

* adding schema files to the zip download

* revert yarn change

* fixing list

* indentation

* 124 summary csv files (#175)

* aAdding summary csv reports

* adding comments

* fixing off by 1 error in summary counts (#177)

* Consistent csv delimiter (#179)

* Only using comma delimited for summary csv files and automatically quoting all string fields in the csv

* Simplifying generateSummaryFiles and removing array mutation

* fixing unit test

* only run typescript tests with test-ts

* Updated babel and dev package to run tests

---------

Co-authored-by: Steven Hollingsworth <[email protected]>

* Fixed broken yarn.lock

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Dan Fey <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Zesky665 <[email protected]>
feydan added a commit that referenced this pull request Aug 5, 2023
* Adding a link to Running-man-01/trashai_nbs project

* Bump vm2 from 3.9.17 to 3.9.19 in /infra (#162)

Bumps [vm2](https://github.com/patriksimek/vm2) from 3.9.17 to 3.9.19.
- [Release notes](https://github.com/patriksimek/vm2/releases)
- [Changelog](https://github.com/patriksimek/vm2/blob/master/CHANGELOG.md)
- [Commits](patriksimek/vm2@3.9.17...3.9.19)

---
updated-dependencies:
- dependency-name: vm2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump vite from 3.2.5 to 3.2.7 in /infra (#165)

Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 3.2.5 to 3.2.7.
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/v3.2.7/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/v3.2.7/packages/vite)

---
updated-dependencies:
- dependency-name: vite
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump vite from 3.0.4 to 3.2.7 in /frontend (#164)

Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 3.0.4 to 3.2.7.
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/v3.2.7/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/v3.2.7/packages/vite)

---
updated-dependencies:
- dependency-name: vite
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fixing build and fixing local (#170)

* Add testing instructions and auto tests (#171)

* Adding a link to Running-man-01/trashai_nbs project (#168)

* update frontend app doc

* Add manual instructions.

* Added test-ids

* Add test-ids to the navigation tabs

* Add smoke test draft

* Add placeholder file

* Add photos for manual testing instructions

* Upgrade node version

* Remove cypress

* Add test ids to summary page

* Jest + Puppeteer setup

* Add smoke test

* Added back manual instructions

* Add images for manual instructions

* Add instruction for running auto tests

---------

Co-authored-by: Dan Fey <[email protected]>

* Adding schema files to download (#173)

* adding schema files to the zip download

* revert yarn change

* fixing list

* indentation

* 124 summary csv files (#175)

* aAdding summary csv reports

* adding comments

* fixing off by 1 error in summary counts (#177)

* Consistent csv delimiter (#179)

* Only using comma delimited for summary csv files and automatically quoting all string fields in the csv

* Simplifying generateSummaryFiles and removing array mutation

* fixing unit test

* only run typescript tests with test-ts

* Updated babel and dev package to run tests

---------

Co-authored-by: Steven Hollingsworth <[email protected]>

* Fixed broken yarn.lock

* Review1 (#181)

* my workflow

* example templates for manuscript.

* manuscript info

* update citations

* cleanup text

* cite

* update readme

* contribute

* add website badge

* update web link

* Update README.md

* Update README.md

* add video link

* auto update

* utility files device specific

* move the pdf generator to workflows

* test error in paper.bib

* test citation style

* add citations

* add a few more citations

* try without space

* add all

* no spaces allowed in citation names

* add demo images

* Update paper.md

* give images some space

* image captions

* add mention of wade

* update authors and acknowledgements

* Updated with Dan's Recs

* Mary Comments

#69 (comment)

* SteveO comments

#69 (review)

* Walter's comments

#69 (review)

* Create config.yml

* Create bug.yml

* Create feature.yml

* hyperlink

* Update bug.yml

* Update feature.yml

* Added in Kristiina and Kris's comments

* Update README.md

* #69 (comment)

* #69 (comment)

* #69 (comment)

* #69 (comment)

* #69 (comment)

* #69 (comment)

* add kris's comments

* add WSL2 link

* Update paper.md

* add acknowledgements

* #69 (comment)

* #69 (comment)

* #69 (comment)

* add submitted badge

* update dois

* Update paper.md

#122

* Update paper.md

add Elizabeth to ack

* Update paper.md

add funder possibility lab

* Update paper.md

add funder.

* Update README.md

* add r code for analyzing data

* remove unnecessary code.

* trying to fix the unexpected period issue, not sure where it is coming from.

* revert bib to test

* add dois

* update paper acknowledgments and links.

* add comments.

* remove empty line

* Update about.vue

update about

* Update about.vue

add tutorial

* Update README.md

update video

* Update paper.md

update video

* Bump word-wrap from 1.2.3 to 1.2.4 in /frontend (#187)

Bumps [word-wrap](https://github.com/jonschlinkert/word-wrap) from 1.2.3 to 1.2.4.
- [Release notes](https://github.com/jonschlinkert/word-wrap/releases)
- [Commits](jonschlinkert/word-wrap@1.2.3...1.2.4)

---
updated-dependencies:
- dependency-name: word-wrap
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump word-wrap from 1.2.3 to 1.2.4 in /infra (#186)

Bumps [word-wrap](https://github.com/jonschlinkert/word-wrap) from 1.2.3 to 1.2.4.
- [Release notes](https://github.com/jonschlinkert/word-wrap/releases)
- [Commits](jonschlinkert/word-wrap@1.2.3...1.2.4)

---
updated-dependencies:
- dependency-name: word-wrap
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump semver from 5.7.1 to 5.7.2 in /frontend (#185)

Bumps [semver](https://github.com/npm/node-semver) from 5.7.1 to 5.7.2.
- [Release notes](https://github.com/npm/node-semver/releases)
- [Changelog](https://github.com/npm/node-semver/blob/v5.7.2/CHANGELOG.md)
- [Commits](npm/node-semver@v5.7.1...v5.7.2)

---
updated-dependencies:
- dependency-name: semver
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump semver from 5.7.1 to 5.7.2 in /infra (#184)

Bumps [semver](https://github.com/npm/node-semver) from 5.7.1 to 5.7.2.
- [Release notes](https://github.com/npm/node-semver/releases)
- [Changelog](https://github.com/npm/node-semver/blob/v5.7.2/CHANGELOG.md)
- [Commits](npm/node-semver@v5.7.1...v5.7.2)

---
updated-dependencies:
- dependency-name: semver
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump tough-cookie from 4.1.2 to 4.1.3 in /infra (#182)

Bumps [tough-cookie](https://github.com/salesforce/tough-cookie) from 4.1.2 to 4.1.3.
- [Release notes](https://github.com/salesforce/tough-cookie/releases)
- [Changelog](https://github.com/salesforce/tough-cookie/blob/master/CHANGELOG.md)
- [Commits](salesforce/tough-cookie@v4.1.2...v4.1.3)

---
updated-dependencies:
- dependency-name: tough-cookie
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump semver from 5.7.1 to 5.7.2 in /backend (#183)

Bumps [semver](https://github.com/npm/node-semver) from 5.7.1 to 5.7.2.
- [Release notes](https://github.com/npm/node-semver/releases)
- [Changelog](https://github.com/npm/node-semver/blob/v5.7.2/CHANGELOG.md)
- [Commits](npm/node-semver@v5.7.1...v5.7.2)

---
updated-dependencies:
- dependency-name: semver
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Zesky665 <[email protected]>
Co-authored-by: Steven Hollingsworth <[email protected]>
Co-authored-by: Win Cowger, PhD <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants