Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Windows Compatibility and Smaller Tiles #8

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

GlassesNerd
Copy link

@GlassesNerd GlassesNerd commented Jul 30, 2023

For Windows Compatibilty

  1. In hyper_dataloader.py, changed the file paths to use os.path.join() instead of directly using "/" so that windows file path is also compatible.
  2. Code still works as original on Linux/MacOS.
  3. Added instructions in methanemapper/README.md to create a junction in windows to link the dataset folder
  4. Fixed some typos ("No create soft links" -> "Now create soft links") (linking command had wrong argument, "train_dataset" folder does not exist)
  5. Added .bat equivalent of .sh files for convenient running on windows

For 128px*128px tiles

  1. Added command line argument small_tiles to indicate whether to use 128px tiles or 256px tiles
  2. 128px tiles are joined with np.concatenate() to recreate original 256px tiles
  3. used npz compression in addition to splitting the tiles to further decrease dataset size
  4. 2020 mhs dataset size reduced from 1.07TB to 228GB
  5. Training time increases about 10% on SSD, 20% on HDD

@GlassesNerd GlassesNerd changed the title Windows Compatibility Windows Compatibility and Smaller Tiles Aug 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant