Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated trainer lifecycle #87

Merged
merged 1 commit into from
Dec 21, 2022

Conversation

orperel
Copy link
Collaborator

@orperel orperel commented Dec 21, 2022

Contains updates to the trainer lifecycle and better docs for tracers:

  • After changing trainer to "steps by iterations" rather than by epochs, the gui step counter was bugged - this is fixed by this MR.
  • Added pre_train / post_train functions which are useful for logging (i.e. shutting off the wandb connection)
  • Some logging logic (mostly wandb) shifted to the trainer
  • A fix to df.parquet which couldn't dump torch tensors
  • Some extra args in trainer are handled more carefully to avoid crashes for the upcoming reformatted config parser

Signed-off-by: operel [email protected]

Restore default ctor with docs

Signed-off-by: operel <[email protected]>
@orperel orperel merged commit 240cee0 into NVIDIAGameWorks:main Dec 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant