Add config file schema

[adamjstewart]

Summary

We should add a schema for our config files.

Rationale

We recently changed our config file layout. However, all old scripts continue to run without error, even though most of the keys are being ignored. We should enforce valid config files to avoid this issue.

Implementation

Hydra has a way to do this: https://hydra.cc/docs/tutorials/structured_config/schema/

The tricky part is keys like pl.Trainer where the valid keys change frequently and we don't have easy access to them. We'll have to see if we can ignore them and let the object instantiation handle that. Maybe we only validate the top-level keys and skip all objects?

Alternatives

No response

Additional information

No response

[mayeroa]

Hi,

This is a great idea. Enforcing the config to be validated against some schema may help to reduce some errors.
I would also recommend using https://github.com/mit-ll-responsible-ai/hydra-zen which may help to handle dynamic creation of structured configs and reduce boilerplate. It could be an option as well. :)

[adamjstewart]

It isn't clear to me what hydra-zen offers. It seems like their goal is removing YAML files. But we already support both YAML and CLI and same hyperparameters.