Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consistent csv delimiter #179

Merged
merged 5 commits into from
Jul 1, 2023

Conversation

feydan
Copy link
Member

@feydan feydan commented Jul 1, 2023

Changes

  1. Only using comma to delimit csv summary files
  2. Automatically quoting all string fields in csv file

Purpose

Csv files contained inconsistent delimiters, this makes them all use comma

Approach

Some specific columns in the csv are themselves comma separated, hence the difference in delimeter. However this is not an issue if we instead quote the field properly. This approach now quotes all string fields, which should be safer and more consistent in the long run in case there are other use cases added to generate csv files.

Testing

  1. Show samples or upload files
  2. Download all files
  3. Inspect the summary*.csv files

@feydan feydan changed the base branch from production to aws/trashai-staging July 1, 2023 15:50
@shollingsworth
Copy link
Collaborator

[1] % npm run test

> [email protected] test
> jest -c jest-puppeteer.config.js

Determining test suites to run...Setup Puppeteer
DIR /var/folders/g9/6b9508h57dq3f4rjgv82mwww0000gp/T/jest_puppeteer_global_setup
Setup Test Environment.
Setup Test Environment.
 PASS  src/lib/csv/schema-parser.spec.ts
Teardown Test Environment.
 PASS  __tests__/smokeTest.js (6.997 s)
Teardown Test Environment.
A worker process has failed to exit gracefully and has been force exited. This is likely caused by tests leaking due to improper teardown. Try running with --detectOpenHandles to find leaks. Active timers can also cause this, ensure that .unref() was called on them.

Test Suites: 2 passed, 2 total
Tests:       5 passed, 5 total
Snapshots:   0 total
Time:        7.904 s, estimated 13 s
Ran all test suites.
Teardown Puppeteer
DIR /var/folders/g9/6b9508h57dq3f4rjgv82mwww0000gp/T/jest_puppeteer_global_setup

shollingsworth@Stevens-MacBook-Pro ‹ consistent-csv-delimiter ● › : ~/Documents/GitHub/trash-ai/frontend
[0] % npm run test-ts

> [email protected] test-ts
> jest

 PASS  src/lib/csv/schema-parser.spec.ts
  csv
    ✓ getCsvHeaders returns the proper headers (2 ms)
    ✓ objToCsv correctly converts an object to a data string (1 ms)
    ✓ objToCsv throws if row lengths are invalid (5 ms)

Test Suites: 1 passed, 1 total
Tests:       3 passed, 3 total
Snapshots:   0 total
Time:        1.345 s, estimated 3 s
Ran all test suites.

Copy link
Collaborator

@shollingsworth shollingsworth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, made some minor changes to package.json and babel.

@shollingsworth shollingsworth merged commit 0fe7dab into aws/trashai-staging Jul 1, 2023
@shollingsworth shollingsworth deleted the consistent-csv-delimiter branch July 1, 2023 20:04
feydan added a commit that referenced this pull request Jul 1, 2023
* Adding a link to Running-man-01/trashai_nbs project

* Bump vm2 from 3.9.17 to 3.9.19 in /infra (#162)

Bumps [vm2](https://github.com/patriksimek/vm2) from 3.9.17 to 3.9.19.
- [Release notes](https://github.com/patriksimek/vm2/releases)
- [Changelog](https://github.com/patriksimek/vm2/blob/master/CHANGELOG.md)
- [Commits](patriksimek/vm2@3.9.17...3.9.19)

---
updated-dependencies:
- dependency-name: vm2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump vite from 3.2.5 to 3.2.7 in /infra (#165)

Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 3.2.5 to 3.2.7.
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/v3.2.7/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/v3.2.7/packages/vite)

---
updated-dependencies:
- dependency-name: vite
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump vite from 3.0.4 to 3.2.7 in /frontend (#164)

Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 3.0.4 to 3.2.7.
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/v3.2.7/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/v3.2.7/packages/vite)

---
updated-dependencies:
- dependency-name: vite
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fixing build and fixing local (#170)

* Add testing instructions and auto tests (#171)

* Adding a link to Running-man-01/trashai_nbs project (#168)

* update frontend app doc

* Add manual instructions.

* Added test-ids

* Add test-ids to the navigation tabs

* Add smoke test draft

* Add placeholder file

* Add photos for manual testing instructions

* Upgrade node version

* Remove cypress

* Add test ids to summary page

* Jest + Puppeteer setup

* Add smoke test

* Added back manual instructions

* Add images for manual instructions

* Add instruction for running auto tests

---------

Co-authored-by: Dan Fey <[email protected]>

* Adding schema files to download (#173)

* adding schema files to the zip download

* revert yarn change

* fixing list

* indentation

* 124 summary csv files (#175)

* aAdding summary csv reports

* adding comments

* fixing off by 1 error in summary counts (#177)

* Consistent csv delimiter (#179)

* Only using comma delimited for summary csv files and automatically quoting all string fields in the csv

* Simplifying generateSummaryFiles and removing array mutation

* fixing unit test

* only run typescript tests with test-ts

* Updated babel and dev package to run tests

---------

Co-authored-by: Steven Hollingsworth <[email protected]>

* Fixed broken yarn.lock

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Dan Fey <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Zesky665 <[email protected]>
feydan added a commit that referenced this pull request Aug 5, 2023
* Adding a link to Running-man-01/trashai_nbs project

* Bump vm2 from 3.9.17 to 3.9.19 in /infra (#162)

Bumps [vm2](https://github.com/patriksimek/vm2) from 3.9.17 to 3.9.19.
- [Release notes](https://github.com/patriksimek/vm2/releases)
- [Changelog](https://github.com/patriksimek/vm2/blob/master/CHANGELOG.md)
- [Commits](patriksimek/vm2@3.9.17...3.9.19)

---
updated-dependencies:
- dependency-name: vm2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump vite from 3.2.5 to 3.2.7 in /infra (#165)

Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 3.2.5 to 3.2.7.
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/v3.2.7/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/v3.2.7/packages/vite)

---
updated-dependencies:
- dependency-name: vite
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump vite from 3.0.4 to 3.2.7 in /frontend (#164)

Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 3.0.4 to 3.2.7.
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/v3.2.7/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/v3.2.7/packages/vite)

---
updated-dependencies:
- dependency-name: vite
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fixing build and fixing local (#170)

* Add testing instructions and auto tests (#171)

* Adding a link to Running-man-01/trashai_nbs project (#168)

* update frontend app doc

* Add manual instructions.

* Added test-ids

* Add test-ids to the navigation tabs

* Add smoke test draft

* Add placeholder file

* Add photos for manual testing instructions

* Upgrade node version

* Remove cypress

* Add test ids to summary page

* Jest + Puppeteer setup

* Add smoke test

* Added back manual instructions

* Add images for manual instructions

* Add instruction for running auto tests

---------

Co-authored-by: Dan Fey <[email protected]>

* Adding schema files to download (#173)

* adding schema files to the zip download

* revert yarn change

* fixing list

* indentation

* 124 summary csv files (#175)

* aAdding summary csv reports

* adding comments

* fixing off by 1 error in summary counts (#177)

* Consistent csv delimiter (#179)

* Only using comma delimited for summary csv files and automatically quoting all string fields in the csv

* Simplifying generateSummaryFiles and removing array mutation

* fixing unit test

* only run typescript tests with test-ts

* Updated babel and dev package to run tests

---------

Co-authored-by: Steven Hollingsworth <[email protected]>

* Fixed broken yarn.lock

* Review1 (#181)

* my workflow

* example templates for manuscript.

* manuscript info

* update citations

* cleanup text

* cite

* update readme

* contribute

* add website badge

* update web link

* Update README.md

* Update README.md

* add video link

* auto update

* utility files device specific

* move the pdf generator to workflows

* test error in paper.bib

* test citation style

* add citations

* add a few more citations

* try without space

* add all

* no spaces allowed in citation names

* add demo images

* Update paper.md

* give images some space

* image captions

* add mention of wade

* update authors and acknowledgements

* Updated with Dan's Recs

* Mary Comments

#69 (comment)

* SteveO comments

#69 (review)

* Walter's comments

#69 (review)

* Create config.yml

* Create bug.yml

* Create feature.yml

* hyperlink

* Update bug.yml

* Update feature.yml

* Added in Kristiina and Kris's comments

* Update README.md

* #69 (comment)

* #69 (comment)

* #69 (comment)

* #69 (comment)

* #69 (comment)

* #69 (comment)

* add kris's comments

* add WSL2 link

* Update paper.md

* add acknowledgements

* #69 (comment)

* #69 (comment)

* #69 (comment)

* add submitted badge

* update dois

* Update paper.md

#122

* Update paper.md

add Elizabeth to ack

* Update paper.md

add funder possibility lab

* Update paper.md

add funder.

* Update README.md

* add r code for analyzing data

* remove unnecessary code.

* trying to fix the unexpected period issue, not sure where it is coming from.

* revert bib to test

* add dois

* update paper acknowledgments and links.

* add comments.

* remove empty line

* Update about.vue

update about

* Update about.vue

add tutorial

* Update README.md

update video

* Update paper.md

update video

* Bump word-wrap from 1.2.3 to 1.2.4 in /frontend (#187)

Bumps [word-wrap](https://github.com/jonschlinkert/word-wrap) from 1.2.3 to 1.2.4.
- [Release notes](https://github.com/jonschlinkert/word-wrap/releases)
- [Commits](jonschlinkert/word-wrap@1.2.3...1.2.4)

---
updated-dependencies:
- dependency-name: word-wrap
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump word-wrap from 1.2.3 to 1.2.4 in /infra (#186)

Bumps [word-wrap](https://github.com/jonschlinkert/word-wrap) from 1.2.3 to 1.2.4.
- [Release notes](https://github.com/jonschlinkert/word-wrap/releases)
- [Commits](jonschlinkert/word-wrap@1.2.3...1.2.4)

---
updated-dependencies:
- dependency-name: word-wrap
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump semver from 5.7.1 to 5.7.2 in /frontend (#185)

Bumps [semver](https://github.com/npm/node-semver) from 5.7.1 to 5.7.2.
- [Release notes](https://github.com/npm/node-semver/releases)
- [Changelog](https://github.com/npm/node-semver/blob/v5.7.2/CHANGELOG.md)
- [Commits](npm/node-semver@v5.7.1...v5.7.2)

---
updated-dependencies:
- dependency-name: semver
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump semver from 5.7.1 to 5.7.2 in /infra (#184)

Bumps [semver](https://github.com/npm/node-semver) from 5.7.1 to 5.7.2.
- [Release notes](https://github.com/npm/node-semver/releases)
- [Changelog](https://github.com/npm/node-semver/blob/v5.7.2/CHANGELOG.md)
- [Commits](npm/node-semver@v5.7.1...v5.7.2)

---
updated-dependencies:
- dependency-name: semver
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump tough-cookie from 4.1.2 to 4.1.3 in /infra (#182)

Bumps [tough-cookie](https://github.com/salesforce/tough-cookie) from 4.1.2 to 4.1.3.
- [Release notes](https://github.com/salesforce/tough-cookie/releases)
- [Changelog](https://github.com/salesforce/tough-cookie/blob/master/CHANGELOG.md)
- [Commits](salesforce/tough-cookie@v4.1.2...v4.1.3)

---
updated-dependencies:
- dependency-name: tough-cookie
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump semver from 5.7.1 to 5.7.2 in /backend (#183)

Bumps [semver](https://github.com/npm/node-semver) from 5.7.1 to 5.7.2.
- [Release notes](https://github.com/npm/node-semver/releases)
- [Changelog](https://github.com/npm/node-semver/blob/v5.7.2/CHANGELOG.md)
- [Commits](npm/node-semver@v5.7.1...v5.7.2)

---
updated-dependencies:
- dependency-name: semver
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Zesky665 <[email protected]>
Co-authored-by: Steven Hollingsworth <[email protected]>
Co-authored-by: Win Cowger, PhD <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants