Usage Question: User defined configuration

[jgbos]

I have an usage question. I'm designing a toolbox that will contain a configuration of approaches to train and evaluate ML models. This toolbox is an application that can be installed with it's own configuration files.

Is there a way for the user to:

• Define their own set of configuration files locally: configs/<groups>/user_impl.yaml. For example, their own dataset and models
• Define their own local config.yaml file that can utilize the local configurations and the packaged configurations

I'm struggling to figure out a solution to this. Do I use the search_path plugin? The compose interface? Structured configs? I may be missing something simple. I feel like it's right there but I can't see it!

[jgbos]

I am able to do the OmegaConf.merge approach in #274, but that has some limitations on the interpolation. I'm not unsure how to implement the search path plugin you recommend. I'll keep trying as this won't be high priority for you.

[omry]

Hi @jgbos.
Take a look at the config path plugin:
https://github.com/facebookresearch/hydra/tree/master/plugins/examples/example_searchpath_plugin

This is the right way to achieve what you are after.

Users can add their own configs in the config_path specified in their @hydra.main().

Keep in mind that currently the composition order will not be what you want with respect to the primary config file:

config.yaml:

defaults:
  - hydra/launcher: foo
  - framework/foo : bar

config_in_config_py: oompa

the composition order will be:

• hydra/launcher (all of Hydra's configurations really)
• config.yaml
• anything else in the defaults.

This means the user will not be able to override your framework configs from their config.yaml.
to solve it, the user can do:

defaults.yaml

defaults:
  - hydra/launcher: foo
  - framework/foo : bar
  - config

config.yaml

config_in_config_py: oompa

@hydra.main(config_path="conf", config_name="defaults")
def main(cfg : DictConfig): 
  ...

if __name__ == "__main__": 
   main()

[omry]

Also, if you are not already - please us Hydra 1.0.0rc1 or master for this.

[jgbos]

Thanks for taking the time tor respond to this questions. I think I'm getting closer to understanding the concept but currently still getting errors, essentially it cannot find framework/foo. Does it matter where I put the search path plugin code in the framework?

from hydra.core.config_search_path import ConfigSearchPath
from hydra.plugins.search_path_plugin import SearchPathPlugin

class FrameworkSearchPathPlugin(SearchPathPlugin):
    def manipulate_search_path(self, search_path: ConfigSearchPath):
        search_path.append(
            "framework",
            "pkg://framework.configs",
        )

[omry]

It does matter, yes.
It has to in a top level hydra_plugins module, just like in the example.
You can run your app with --info to see the detected plugins and the search path.

[jgbos]

OK! I see now what I wasn't seeing before (I hope). I was trying to just put the code in my framework, but I need to make a search path plugin for my framework for hydra to find it. Is this correct? I create a plugin directory with hydra_plugins/framework_plugin and follow your setup.py file for example searchpath plugin, but I don't see the plugin when I do --info.

As test, I cloned hydra and in the example_searchpath_plugin directory I did python setup.py install and I cannot see the plugin for this example either.

[omry]

The example plugin is working as has unit tests.
can you install it and run pytest on that directory?

[omry]

it's probably not the reason, but use pip install ., not setup.py directly.

[jgbos]

Ack. That was it! I had to do pip install . 🙄

Thanks for your patience and help, much appreciated

[omry]

Awesome.
one last piece of advice:

Once you are exporting your configs via a searchpath plugin, you are in the same config namespace as other projects.
but sure to put your configs under a sub directory to avoid nasty collisions down the line.

example:

configs/
   mytoolkit/
     group1/
        option1.yaml
        option2.yaml

along with

class FrameworkSearchPathPlugin(SearchPathPlugin):
    def manipulate_search_path(self, search_path: ConfigSearchPath):
        search_path.append(
            "framework",
            "pkg://framework.configs",
        )

[jgbos]

One question, you say above the defaults are set with

defaults:
  - hydra/launcher: foo
  - framework/foo : bar
  - config 

but I had do to:

defaults:
  - hydra/launcher: basic
  - foo : user_defined  # foo is defined in `framework` but has a user defined foo in conf/foo/user.yaml'> 

It doesn't recognize framework/foo

[jgbos]

I didn't see the previous comment when typing that out.

[omry]

1. no need to specify hydra/launcher, this was just to illustrate the point. you can remove it.
2. feel free to join the chat if you have questions.