From 19621a8c10949d2a88192f934868af14419dcded Mon Sep 17 00:00:00 2001 From: Dan Fey Date: Wed, 12 Oct 2022 21:06:43 -0700 Subject: [PATCH] Bump prod from staging (#83) * Coco POC #8 #9 (#19) (#26) * Fixed some package.json issues, updated some entries in .gitignore and .in (for those who would use it) * saving work * saving work... having issues with async reading file then loading it in coco * coco confirmed working * Added JSON metadata * Arg! this should work * Removed Pro port mapping from localstack docker-compose * Added multiple file upload and cleaned up presentation a bit Looks like there is a problem with uploading multiple files at once, the predictions don't always trigger * Minor updates to MacOS details (#24) Co-authored-by: Jim Co-authored-by: Jim Ewald Co-authored-by: Jim Co-authored-by: Jim Ewald Co-authored-by: Jim * removed symlink * Testing coco (#28) * Fixed some package.json issues, updated some entries in .gitignore and .in (for those who would use it) * saving work * saving work... having issues with async reading file then loading it in coco * coco confirmed working * Added JSON metadata * Arg! this should work * Removed Pro port mapping from localstack docker-compose * Added multiple file upload and cleaned up presentation a bit Looks like there is a problem with uploading multiple files at once, the predictions don't always trigger * Minor updates to MacOS details (#24) Co-authored-by: Jim * removed symlink to web_model * Coco POC #8 #9 (#19) (#26) * Fixed some package.json issues, updated some entries in .gitignore and .in (for those who would use it) * saving work * saving work... having issues with async reading file then loading it in coco * coco confirmed working * Added JSON metadata * Arg! this should work * Removed Pro port mapping from localstack docker-compose * Added multiple file upload and cleaned up presentation a bit Looks like there is a problem with uploading multiple files at once, the predictions don't always trigger * Minor updates to MacOS details (#24) Co-authored-by: Jim Co-authored-by: Jim Ewald Co-authored-by: Jim Co-authored-by: Jim Ewald Co-authored-by: Jim * removed symlink Co-authored-by: Jim Ewald Co-authored-by: Jim * Changed upload behaviour * Refactored upload dialog / drop area * added more model samples * removed dropzone package * Fixed issue with remove / key and array order * Removed lab / test for deployed versions of environment * Added about page * Forgot to update the metadata text * Yolov5-Taco web model #7 (#30) * Testing tf_web model export * removing pyright * Working version of taco dataset + initial build instructions * Created links to code4sac and the github project Added jupyter notebook files for new training as well as using existing training data * Adjusted new training jupyter notebook * Testing change to model loading in AWS * Try 2 on loading model * Try 3 on loading model * Reverting changes (has to do with basic auth url) * Removed old react frontend directory * Removed old react frontend directory (#34) * Moving domain from codefordev to trashai.org * Replaced Backend with python, fixed infa deploy logic # 35 * backend now tracks images by their sha256 value to avoid duplication * added new manual action to infra deploy logic # 35 * added log retention logic to backend deploy # 36 * added new fields to deploy_map "log_retention_days": 30, "dns_domain_map_root": true, * made github_actions script aware of domain, and "is_root_domain" deploy_map setting * Addresses # 32, added new features amongst them a download all button * added new packages to UI * "dexie" local storage w/ indexeddb browser store, localstore couldn't cut it * "file-saver": allows us to do a saveas with the zip file * "jszip": zip libarary to address # * indicators for backend upload success, and if the image has been seen before * now looks good in mobile as well as desktop * also using indexeddb introduced caching the file objects (for download later) and caching of the tf model * Added support for python packages * Adjusted permission to add/remove layers in the prefix namespace * fixed permissions issue for layers / backend deploy * Added hash and other metadata to the metadata display and hover over filename * Fixed metadata download and info button * Fixes for #36, #35, #32 + download all feature, and backend refactor w/ dedup (#38) * Replaced Backend with python, fixed infa deploy logic # 35 * backend now tracks images by their sha256 value to avoid duplication * added new manual action to infra deploy logic # 35 * added log retention logic to backend deploy # 36 * added new fields to deploy_map "log_retention_days": 30, "dns_domain_map_root": true, * made github_actions script aware of domain, and "is_root_domain" deploy_map setting * Addresses # 32, added new features amongst them a download all button * added new packages to UI * "dexie" local storage w/ indexeddb browser store, localstore couldn't cut it * "file-saver": allows us to do a saveas with the zip file * "jszip": zip libarary to address # * indicators for backend upload success, and if the image has been seen before * now looks good in mobile as well as desktop * also using indexeddb introduced caching the file objects (for download later) and caching of the tf model * Added support for python packages * Adjusted permission to add/remove layers in the prefix namespace * fixed permissions issue for layers / backend deploy * Added hash and other metadata to the metadata display and hover over filename * Fixed metadata download and info button * Fixing file ext name issue with s3 bucket upload on backend * bugfix on s3 naming with file extension (#40) * Added magnifying feature for larger images * filling out the about page (#45) * filling out the about page * adding article * better formatting * Made about page default #49 * Added "Samples" button to upload tab closes #50 * Making about updates specified in #48 (#51) * Making about updates specified in #48 * removing space * small about update * Demoing the CI/CD integration * Typescript refactor Version 1 * removed nuxt * removed more files * adjusted build commands * fixed height issue with uploads * re-implemented backend * adjusted frontend deploy stack * Added github secret for google maps api key * Added dockerfile for frontend * added jszip * Fixed android upload gps issue, fixed mobile status truncating issue * removed uneeded logo files * Typescript refactor (#58) * Replaced Backend with python, fixed infa deploy logic # 35 * backend now tracks images by their sha256 value to avoid duplication * added new manual action to infra deploy logic # 35 * added log retention logic to backend deploy # 36 * added new fields to deploy_map "log_retention_days": 30, "dns_domain_map_root": true, * made github_actions script aware of domain, and "is_root_domain" deploy_map setting * Addresses # 32, added new features amongst them a download all button * added new packages to UI * "dexie" local storage w/ indexeddb browser store, localstore couldn't cut it * "file-saver": allows us to do a saveas with the zip file * "jszip": zip libarary to address # * indicators for backend upload success, and if the image has been seen before * now looks good in mobile as well as desktop * also using indexeddb introduced caching the file objects (for download later) and caching of the tf model * Added support for python packages * Adjusted permission to add/remove layers in the prefix namespace * fixed permissions issue for layers / backend deploy * Added hash and other metadata to the metadata display and hover over filename * Fixed metadata download and info button * Fixing file ext name issue with s3 bucket upload on backend * Promote staging updates to prod (#52) * Coco POC #8 #9 (#19) (#26) * Fixed some package.json issues, updated some entries in .gitignore and .in (for those who would use it) * saving work * saving work... having issues with async reading file then loading it in coco * coco confirmed working * Added JSON metadata * Arg! this should work * Removed Pro port mapping from localstack docker-compose * Added multiple file upload and cleaned up presentation a bit Looks like there is a problem with uploading multiple files at once, the predictions don't always trigger * Minor updates to MacOS details (#24) Co-authored-by: Jim Co-authored-by: Jim Ewald Co-authored-by: Jim Co-authored-by: Jim Ewald Co-authored-by: Jim * removed symlink * Testing coco (#28) * Fixed some package.json issues, updated some entries in .gitignore and .in (for those who would use it) * saving work * saving work... having issues with async reading file then loading it in coco * coco confirmed working * Added JSON metadata * Arg! this should work * Removed Pro port mapping from localstack docker-compose * Added multiple file upload and cleaned up presentation a bit Looks like there is a problem with uploading multiple files at once, the predictions don't always trigger * Minor updates to MacOS details (#24) Co-authored-by: Jim * removed symlink to web_model * Coco POC #8 #9 (#19) (#26) * Fixed some package.json issues, updated some entries in .gitignore and .in (for those who would use it) * saving work * saving work... having issues with async reading file then loading it in coco * coco confirmed working * Added JSON metadata * Arg! this should work * Removed Pro port mapping from localstack docker-compose * Added multiple file upload and cleaned up presentation a bit Looks like there is a problem with uploading multiple files at once, the predictions don't always trigger * Minor updates to MacOS details (#24) Co-authored-by: Jim Co-authored-by: Jim Ewald Co-authored-by: Jim Co-authored-by: Jim Ewald Co-authored-by: Jim * removed symlink Co-authored-by: Jim Ewald Co-authored-by: Jim * Changed upload behaviour * Refactored upload dialog / drop area * added more model samples * removed dropzone package * Fixed issue with remove / key and array order * Removed lab / test for deployed versions of environment * Added about page * Forgot to update the metadata text * Yolov5-Taco web model #7 (#30) * Testing tf_web model export * removing pyright * Working version of taco dataset + initial build instructions * Created links to code4sac and the github project Added jupyter notebook files for new training as well as using existing training data * Adjusted new training jupyter notebook * Testing change to model loading in AWS * Try 2 on loading model * Try 3 on loading model * Reverting changes (has to do with basic auth url) * Removed old react frontend directory * Removed old react frontend directory (#34) * Moving domain from codefordev to trashai.org * Fixes for #36, #35, #32 + download all feature, and backend refactor w/ dedup (#38) * Replaced Backend with python, fixed infa deploy logic # 35 * backend now tracks images by their sha256 value to avoid duplication * added new manual action to infra deploy logic # 35 * added log retention logic to backend deploy # 36 * added new fields to deploy_map "log_retention_days": 30, "dns_domain_map_root": true, * made github_actions script aware of domain, and "is_root_domain" deploy_map setting * Addresses # 32, added new features amongst them a download all button * added new packages to UI * "dexie" local storage w/ indexeddb browser store, localstore couldn't cut it * "file-saver": allows us to do a saveas with the zip file * "jszip": zip libarary to address # * indicators for backend upload success, and if the image has been seen before * now looks good in mobile as well as desktop * also using indexeddb introduced caching the file objects (for download later) and caching of the tf model * Added support for python packages * Adjusted permission to add/remove layers in the prefix namespace * fixed permissions issue for layers / backend deploy * Added hash and other metadata to the metadata display and hover over filename * Fixed metadata download and info button * bugfix on s3 naming with file extension (#40) * Added magnifying feature for larger images * filling out the about page (#45) * filling out the about page * adding article * better formatting * Made about page default #49 * Added "Samples" button to upload tab closes #50 * Making about updates specified in #48 (#51) * Making about updates specified in #48 * removing space * small about update Co-authored-by: Steven Hollingsworth Co-authored-by: Jim Ewald Co-authored-by: Jim * Typescript refactor Version 1 * removed nuxt * removed more files * adjusted build commands * fixed height issue with uploads * re-implemented backend * adjusted frontend deploy stack * Added github secret for google maps api key * Added dockerfile for frontend * added jszip * Fixed android upload gps issue, fixed mobile status truncating issue * removed uneeded logo files Co-authored-by: Dan Fey Co-authored-by: Jim Ewald Co-authored-by: Jim * updated deploy scripts * testing config update * added vite * testing vite * testing vite * testing deploy * adjusted permissions to allow access to public bucket * still testing * arg! * Updating about page (#64) * Updating about page to show Steve as the lead dev and adding feedback form * making about pages consistent * adding google analytics script (#67) * removing kaggle depedency on running model notebook (#82) Co-authored-by: Steven Hollingsworth Co-authored-by: Jim Ewald Co-authored-by: Jim --- model/build/README.md | 8 -------- model/build/docker-compose.yaml | 22 ++++++++++------------ model/build/run_notebook.sh | 7 ------- 3 files changed, 10 insertions(+), 27 deletions(-) diff --git a/model/build/README.md b/model/build/README.md index a32f1c3..457c116 100644 --- a/model/build/README.md +++ b/model/build/README.md @@ -1,11 +1,3 @@ -# Prerequisites -Create a kaggle account, and download an API key - - -# Build Instructions - -If you run this on kaggle, you'll likely run out of credits after a few runs - # Local build ``` ./run_notebook.sh diff --git a/model/build/docker-compose.yaml b/model/build/docker-compose.yaml index 14b4c41..9e589f2 100644 --- a/model/build/docker-compose.yaml +++ b/model/build/docker-compose.yaml @@ -15,21 +15,19 @@ services: # devices: # - capabilities: [gpu] - dltacodataset: - image: kaggledl + tacodataset: + image: alpine volumes: - - ${HOME}/.kaggle:/home/kaggle/.kaggle - - ${PWD}:/home/kaggle/download - working_dir: /home/kaggle/download - command: kaggle datasets download -d kneroma/tacotrashdataset + - ${PWD}:/download + working_dir: /download + command: wget https://trash-ai-public.s3.us-west-1.amazonaws.com/tacotrashdataset.zip - dlyolov5-taco: - image: kaggledl + yolov5-taco: + image: alpine volumes: - - ${HOME}/.kaggle:/home/kaggle/.kaggle - - ${PWD}:/home/kaggle/download - working_dir: /home/kaggle/download - command: kaggle datasets download -d trngvhong/yolov5-taco + - ${PWD}:/download + working_dir: /download + command: wget https://trash-ai-public.s3.us-west-1.amazonaws.com/yolov5-taco.zip notebook: image: jupyter/datascience-notebook:b418b67c225b diff --git a/model/build/run_notebook.sh b/model/build/run_notebook.sh index 3f92901..d519231 100755 --- a/model/build/run_notebook.sh +++ b/model/build/run_notebook.sh @@ -2,12 +2,5 @@ set -euo pipefail IFS=$'\n\t' -mkdir -p ~/.kaggle -test -f ~/.kaggle/kaggle.json || { - echo "Kaggle API key not found. Please login and download api key and put it in ~/.kaggle/kaggle.json" - exit 1 -} -chmod 600 ~/.kaggle/kaggle.json - docker-compose up --build ./name_map.py