Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use DL1DataHandler for data loading #115

Closed
11 of 12 tasks
aribrill opened this issue Apr 26, 2019 · 1 comment · Fixed by #117
Closed
11 of 12 tasks

Use DL1DataHandler for data loading #115

aribrill opened this issue Apr 26, 2019 · 1 comment · Fixed by #117
Assignees

Comments

@aribrill
Copy link
Collaborator

aribrill commented Apr 26, 2019

Replace the classes currently used for data loading (DataLoader, DataProcessor, and ImageMapper) with the updated version in the DL1DataHandler package. This package was designed to fix the problems noted in and so resolve #82. It also fixes #46 and closes #73 as no longer needed. This issue involves the follow steps:

  • Remove data_loading.py, data_processor.py, image_mapping.py, and pixel_pos_files, which have been superseded by DL1DH.
  • Remove all scripts and notebooks that exclusively use or are designed to test these modules and create issues in the DL1DH repository to recreate them there as needed.
  • Update run_model.py to use DL1DataReader instead of DataLoader and DataProcessor to load and process the data, specifically using the new DL1DataReader.example_description to more cleanly define the input_fn (Define API for accessing DataLoader class attributes #46).
  • Update run_model.py to use the capability of DL1DataReader to load any telescope and event info to:
    • Load arbitrary auxiliary info,
    • Define and calculate the loss for multiple labels, and
    • Simplify making predictions by returning the true label and event ids within the predictions dictionary
  • Update the configuration arguments where needed to match the DL1DH API.
  • Update to reflect these changes:
    • README.md
    • example_config.yml
    • MANIFEST.in
  • Do a benchmark run to confirm that the changes do not affect speed and performance.
@aribrill
Copy link
Collaborator Author

aribrill commented May 7, 2019

I think that calculation of the loss for multiple labels is best handled separately, so I've made another issue for it #118.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants