Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Major upgrade to TF2 (v2.8) #137

Merged
merged 37 commits into from
Mar 31, 2022
Merged

Major upgrade to TF2 (v2.8) #137

merged 37 commits into from
Mar 31, 2022

Conversation

TjarkMiener
Copy link
Member

@TjarkMiener TjarkMiener commented Jan 4, 2022

This is a PR draft for the CTLearn v0.6.0 using Keras with the latest TensorFlow v.2.8.

ToDos:

  • Handle the data loader with Keras functionality. Previous data loader handler input_fn() should be replaced with a Keras batch generator.
  • Upgrade monoscopic models
  • Upgrade stereoscopic models
  • Enable usage of multiple GPUs via tf.distribute.MirroredStrategy
  • Save only the best model checkpoints
  • Calculate the class weights in the init of the DL1DH reader and apply always the class weights for the classification task
  • Upgrade the yaml environment file. Update dependencies like DL1DH and ctapipe to Python 3.8+.
  • Set the reconstruction tasks from command line. Automatise the selection of labels and their transformations.
  • Plot model architecture

ResNets and singleCNNs are upgraded, but not the stereo models yet.
@TjarkMiener TjarkMiener added this to the v0.6.0 milestone Jan 4, 2022
@TjarkMiener TjarkMiener self-assigned this Jan 4, 2022
@TjarkMiener
Copy link
Member Author

I've marked this PR as a draft because there are some things missing. @nietootein regarding the installation I was wondering if we can give it a try to rather install CUDA from the Nvidia conda account in the environment than systemwide? It'd be useful for non-root users.

@TjarkMiener TjarkMiener changed the title Major upgrade to TF2 (v2.7) Major upgrade to TF2 (v2.8) Feb 11, 2022
TjarkMiener and others added 18 commits February 28, 2022 12:46
Reshape image shape to concat telescopes (mainly used in MAGIC)

Added telescope(s) selection from command line. Makes config files per telescope types redundant.

Solved bad file descriptor OSError at the end of training.
Users should use the installed version of the models by default. If they would like to customize the models they can provide the path to their models.
This commit adds several options to the command line in order to keep the config file as general as possible. As a result we can ship generic config files for each model with the installation of CTLearn. Also the user can select the model from the command line without providing an own config file. Developers can still use all the features as before.
Energy labels had shape (batch_size, ), now it is (batch_size, label_shape), as it is for particle type labels and direction labels.
Set energy label shape to (batch_size, label_shape)
@TjarkMiener TjarkMiener marked this pull request as ready for review March 29, 2022 18:19
@TjarkMiener
Copy link
Member Author

Ready from my side!

Some are deprecated due to new functionality. They will keep in the repo for now in case users wants to get inspired by them and write their own scripts.
Copy link
Member

@nietootein nietootein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks so much for this massive upgrade, @TjarkMiener!

@nietootein nietootein merged commit 6198b60 into master Mar 31, 2022
@TjarkMiener TjarkMiener deleted the tf_update branch April 3, 2022 20:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants