Allow users to explicitly pass config files from the command-line.

[erip]

🚀 Feature Request

A user should be able to explicitly pass a config file to use as a command-line argument.

Motivation

Hard-coded paths makes scripts using hydra brittle and confusing (a là the "what's my CWD?" game). A user should be able to pass a config explicitly.

Pitch

Describe the solution you'd like
A config_file command-line flag could be added which, when provided, is the config to be used.

Describe alternatives you've considered

• No command-line flag is added. A user must always ensure the hard-coded config file exists where expected.
• A default config is provided and shipped with hydra-enabled applications and overridden manually.

Are you willing to open a pull request? (See CONTRIBUTING)
Probably. :-)

[hrchu]

+1

[hrchu]

My current workaround is something like this:

def run():
    parser = argparse.ArgumentParser()
    parser.add_argument("CONFIG", help="Config file")
    args = parser.parse_args()
    config_path = args.CONFIG

    run_hydra(
        get_args(version=hydra.__version__),
        task_function,
        config_path,
        strict,
    )

run()

[omry]

Hi,
This is actually a very conscience choice:
With Hydra you can compose an infinite number of configuration from the command line.
I have yet to see a use case that actually justifies allowing an external config file.

I would like to address the "brittle and confusing" part:
in principle, your config should be right next to your code, and should also be committed to the repo.
That config should also be portable and useful for other people.

I get that this is probably different how you are doing things, but I have a concern that you are missing a large part of what Hydra can do, which is likely why you are asking for it.

Can you tell me more about the specific use case that requires multiple configuration files and that cannot be handled by configuration composition?

[erip]

I think it's true to say config should be next to applications, but does it make sense to ship configs with library code? Hydra does this, but should all libraries?

[omry]

I think it does make sense to provide default configs with the library.
Hydra allows your users to override everything. this is in the form of both command line overrides or composition.

Libraries are not well covered by the documentation right now, but I am happy to work with you here and show you what I have in mind for libraries.
In short, a library can add itself to the config search path, thus allowing anyone composing it's configuration files into their own config.
While I don't have good public examples of this yet, I think this is actually a very powerful concept that can make integrating different libraries much easier for the end user - provided that they are using Hydra.

Can you tell me more about your library?
Is it a machine learning framework? (taking a guess)

[erip]

Yes, it will be a machine learning library which will have configuration that will make integration with hydra super convenient. There are various configs for e.g., tasks, hyperparameter options, data, etc.

[omry]

Sounds great.
I am happy to answer any questions you have about integrating Hydra into such a framework.
we can also switch to messenger if it helps.

[omry]

I am going to close this. at this time I do not see a good enough reason to introduce this mechanism. my primary concern is that once this is there people would use it instead of composition, which would defeat the purpose of using Hydra.
I am still open to discussion and am happy to discuss specific use cases, both to educate what is the right way to handle them with Hydra and possible to concede that there is actually a good reason to support this functionality.

[hrchu]

@omry how do you think about UNIX convention, i.e., configs in /etc and ~/.config?

[omry]

@hrchu, I think it might make sense to allow adjusting the config search path to support external configuration, but even so the right solution would probably not be to replace the main config file but just allow composing from sources in other directories.
can you tell me more about a concrete use case where you would need to have system or user specific configuration?

[aosokin]

Hi, is this still alive?
I think that in machine learning applications using external config files is very handy.
Say I have my training method with the default config for everything.
Then, I'm doing multiple rounds of experiments.
For each round, I want to define many config parameters identically but vary just one.
It is very inconvenient to set up a whole bunch of non-default hyper-parameters in many launching scripts. I'd prefer to have one external (partial) config file with non-default parameters for each round and to define only the varying parameters through command line.

[omry]

Still alive in #386.
I am going to comment there with a clean workaround for many of the use cases.