From 0df4837cefd6618c6a3fe872b98e4f79791c2c4a Mon Sep 17 00:00:00 2001 From: Jieru Hu Date: Tue, 8 Dec 2020 14:13:21 -0800 Subject: [PATCH 1/3] fix configerator plugin doc. --- website/docs/fb/configerator_config_source.md | 37 +++++++++++++++++++ website/docs/fb/fbcode.md | 1 + 2 files changed, 38 insertions(+) create mode 100644 website/docs/fb/configerator_config_source.md diff --git a/website/docs/fb/configerator_config_source.md b/website/docs/fb/configerator_config_source.md new file mode 100644 index 00000000000..5dac1d1f003 --- /dev/null +++ b/website/docs/fb/configerator_config_source.md @@ -0,0 +1,37 @@ +--- +id: fbcode-configerator-config-source +title: Configerator Config Source Plugin +--- + +The ConfigeratorConfigSource plugin makes it possible for Hydra applications to consume a domain of configs from configerator. + +### Dependency + +Add the following to your `TARGET` file +```commandline +//fair_infra/fbcode_hydra_plugins/configerator_config_source:configerator_config_source +``` + +### Usage + +1. Since `configerator` does not have a client for listing configs under a path, you need to group your configs under +a `domain` for Hydra to access. + 1. Matching the name of your domain and directory of configs is necessary for the plugin to extract information on the full config names, due to limitations from Configerator APIs + 1. For example, the config paths returned by Configerator API could look like `fair_infra/hydra_plugins/configerator_config_source/example/db/mysql`. The plugin needs to know where the directory of configs begins ([`example`](https://fburl.com/diffusion/7c0c5tig)), in order to determine the full config name (`db/mysql`). So in this case the domain should be named [`example.cconf`](https://fburl.com/diffusion/pyymoo1t) +1. Create a [SearchPathPlugin](https://hydra.cc/docs/next/advanced/search_path) to add the Configerator path to the list of search paths. + 1.The path you add in your SearchPathPlugin should be the name of your domain of configs, such as in this [example SearchPathPlugin](https://fburl.com/diffusion/ljggtux5) + + Note: There will be an easier way to add the Configerator path to the list of search paths, with a planned change in Hydra 1.1 to [allow configuring the search path via the config file](https://github.com/facebookresearch/hydra/issues/274). + +1. Modify the `config_name` in the `@hydra.main` decorator of your application. + 1.To fetch a config file, one way is to modify the `config_name` in the `@hydra.main` decorator of your application should be a logical path to a config within the domain passed to your SearchPathPlugin. + +1. Your config file can contain a [defaults list](https://hydra.cc/docs/next/tutorials/basic/your_first_app/defaults) to select defaults for config groups. To use a defaults list in your config, import the [`configerator/source/fair_infra/hydra_plugins/configerator_config_source/defults`](https://fburl.com/diffusion/xhmy4bwc) thrift structure and specify a default_dict (for defaults in config groups) and/or default_string (for default configs) +- When the application is run, the configs specified in the defaults list are loaded by default. +- For example, [`config.cconf`](https://fburl.com/diffusion/tm9qbpm8) imports the `hydra_configerator_config_default` thrift to create the defaults list, with [`mysql`](https://fburl.com/diffusion/cxgennae) as the default for the [`db`](https://fburl.com/diffusion/99i3uxpu) config group + +### Example: + +An example [application](https://fburl.com/diffusion/pndzq58m) using the ConfigeratorConfigSource plugin and example [SearchPathPlugin](https://fburl.com/diffusion/j86krh3r) are provided. The example app has the decorator `@hydra.main(config_name=…)`, where config_name specifies the path to the configs that will be loaded and printed. + +To run the app, run `buck run //fair_infra/fbcode_hydra_plugins/configerator_config_source/example:my_app` from the fbsource/fbcode directory. diff --git a/website/docs/fb/fbcode.md b/website/docs/fb/fbcode.md index 4954c4f7212..ca53eb8ae52 100644 --- a/website/docs/fb/fbcode.md +++ b/website/docs/fb/fbcode.md @@ -16,6 +16,7 @@ title: Hydra at fbcode ### Facebook specified plugins - fbcode_defaults : Changes configuration defaults to be appropriate for fbcode (e.g: Output directories are in `fbcode/outputs` and `fbcode/multirun`) + - configerator_config_source: See doc for more information. #### TARGETS Hydra includes buck TARGETS you can use in fbcode. In general, if there is TARGET there are two options: From 7666cf6863523d0bc597236918d00a30c82574a2 Mon Sep 17 00:00:00 2001 From: Jieru Hu Date: Tue, 8 Dec 2020 18:28:01 -0800 Subject: [PATCH 2/3] Add configerator config source doc --- website/docs/fb/configerator-config-source.md | 52 +++++++++++++++++++ website/docs/fb/fbcode.md | 2 +- website/sidebars.js | 1 + .../fb/configerator-config-source.md} | 30 ++++++++--- 4 files changed, 76 insertions(+), 9 deletions(-) create mode 100644 website/docs/fb/configerator-config-source.md rename website/{docs/fb/configerator_config_source.md => versioned_docs/version-1.0/fb/configerator-config-source.md} (51%) diff --git a/website/docs/fb/configerator-config-source.md b/website/docs/fb/configerator-config-source.md new file mode 100644 index 00000000000..63a1f3d851f --- /dev/null +++ b/website/docs/fb/configerator-config-source.md @@ -0,0 +1,52 @@ +--- +id: fbcode-configerator-config-source +title: Configerator Config Source Plugin +--- + +The ConfigeratorConfigSource plugin makes it possible for Hydra applications to consume a domain of configs from configerator. + +### Dependency + +Add the following to your `TARGET` file +```commandline +//fair_infra/fbcode_hydra_plugins/configerator_config_source:configerator_config_source +``` + +### Usage + +1. Since `configerator` does not have a client for listing configs under a path, you need to group your configs under +a `domain` for Hydra to access. + 1. Matching the name of your domain and directory of configs is necessary for the plugin to extract information on the full config names, due to limitations from Configerator APIs + 1. For example, the config paths returned by Configerator API could look like `fair_infra/hydra_plugins/configerator_config_source/example/db/mysql`. The plugin needs to know where the directory of configs begins ([`example`](https://fburl.com/diffusion/7c0c5tig)), in order to determine the full config name (`db/mysql`). So in this case the domain should be named [`example.cconf`](https://fburl.com/diffusion/pyymoo1t) +1. Create a [SearchPathPlugin](https://hydra.cc/docs/next/advanced/search_path) to add the Configerator path to the list of search paths. + + 1. The path you add in your SearchPathPlugin should be the name of your domain of configs, such as in this [example SearchPathPlugin](https://fburl.com/diffusion/ljggtux5) + + Note: There will be an easier way to add the Configerator path to the list of search paths, with a planned change in Hydra 1.1 to [allow configuring the search path via the config file](https://github.com/facebookresearch/hydra/issues/274). + +### Example: + +#### Example SearchPathPlugin +[`ConfigeratorExampleSearchPathPlugin`](https://fburl.com/diffusion/vwa82fbg) adds the example configerator domain to the search path of the example applications. + +#### Reading primary config from configerator +This example reads its primary config from configerator [here](https://fburl.com/diffusion/twk3smkj) which has a default list defined. + +```commandline +$ buck run //fair_infra/fbcode_hydra_plugins/configerator_config_source/example/primary_config:my_app +... +Parsing buck files: finished in 1.1 sec +... +{'driver': 'mysql', 'user': 'alau'} +``` + +#### Compose config with configerator +This example reads its primary config from local yaml file `primary_config.yaml` but reads config groups info from configerator. + +```commandline +$ buck run //fair_infra/fbcode_hydra_plugins/configerator_config_source/example/config_group:my_app -- +db=mysql +... +Parsing buck files: finished in 1.1 sec +... +{'foo': 'bar', 'driver': 'mysql', 'user': 'alau'} +``` diff --git a/website/docs/fb/fbcode.md b/website/docs/fb/fbcode.md index ca53eb8ae52..de176c20d62 100644 --- a/website/docs/fb/fbcode.md +++ b/website/docs/fb/fbcode.md @@ -16,7 +16,7 @@ title: Hydra at fbcode ### Facebook specified plugins - fbcode_defaults : Changes configuration defaults to be appropriate for fbcode (e.g: Output directories are in `fbcode/outputs` and `fbcode/multirun`) - - configerator_config_source: See doc for more information. + - configerator_config_source: See [doc](/fb/configerator-config-source.md) for more information. #### TARGETS Hydra includes buck TARGETS you can use in fbcode. In general, if there is TARGET there are two options: diff --git a/website/sidebars.js b/website/sidebars.js index 41cdec4cc8d..a406dd88971 100755 --- a/website/sidebars.js +++ b/website/sidebars.js @@ -158,6 +158,7 @@ module.exports = { 'fb/fbcode', 'fb/internal-fb-cluster', 'fb/fair-cluster', + 'fb/fbcode-configerator-config-source', ]), } } diff --git a/website/docs/fb/configerator_config_source.md b/website/versioned_docs/version-1.0/fb/configerator-config-source.md similarity index 51% rename from website/docs/fb/configerator_config_source.md rename to website/versioned_docs/version-1.0/fb/configerator-config-source.md index 5dac1d1f003..29ceda868e2 100644 --- a/website/docs/fb/configerator_config_source.md +++ b/website/versioned_docs/version-1.0/fb/configerator-config-source.md @@ -23,15 +23,29 @@ a `domain` for Hydra to access. Note: There will be an easier way to add the Configerator path to the list of search paths, with a planned change in Hydra 1.1 to [allow configuring the search path via the config file](https://github.com/facebookresearch/hydra/issues/274). -1. Modify the `config_name` in the `@hydra.main` decorator of your application. - 1.To fetch a config file, one way is to modify the `config_name` in the `@hydra.main` decorator of your application should be a logical path to a config within the domain passed to your SearchPathPlugin. +### Example: -1. Your config file can contain a [defaults list](https://hydra.cc/docs/next/tutorials/basic/your_first_app/defaults) to select defaults for config groups. To use a defaults list in your config, import the [`configerator/source/fair_infra/hydra_plugins/configerator_config_source/defults`](https://fburl.com/diffusion/xhmy4bwc) thrift structure and specify a default_dict (for defaults in config groups) and/or default_string (for default configs) -- When the application is run, the configs specified in the defaults list are loaded by default. -- For example, [`config.cconf`](https://fburl.com/diffusion/tm9qbpm8) imports the `hydra_configerator_config_default` thrift to create the defaults list, with [`mysql`](https://fburl.com/diffusion/cxgennae) as the default for the [`db`](https://fburl.com/diffusion/99i3uxpu) config group +#### Example SearchPathPlugin +[`ConfigeratorExampleSearchPathPlugin`](https://fburl.com/diffusion/vwa82fbg) adds the example configerator domain to the search path of the example applications. -### Example: +#### Reading primary config from configerator +This example reads its primary config from configerator [here](https://fburl.com/diffusion/twk3smkj) which has a default list defined. -An example [application](https://fburl.com/diffusion/pndzq58m) using the ConfigeratorConfigSource plugin and example [SearchPathPlugin](https://fburl.com/diffusion/j86krh3r) are provided. The example app has the decorator `@hydra.main(config_name=…)`, where config_name specifies the path to the configs that will be loaded and printed. +```commandline +$ buck run //fair_infra/fbcode_hydra_plugins/configerator_config_source/example/primary_config:my_app +... +Parsing buck files: finished in 1.1 sec +... +{'driver': 'mysql', 'user': 'alau'} +``` -To run the app, run `buck run //fair_infra/fbcode_hydra_plugins/configerator_config_source/example:my_app` from the fbsource/fbcode directory. +#### Compose config with configerator +This example reads its primary config from local yaml file `primary_config.yaml` but reads config groups info from configerator. + +```commandline +$ buck run //fair_infra/fbcode_hydra_plugins/configerator_config_source/example/config_group:my_app -- +db=mysql +... +Parsing buck files: finished in 1.1 sec +... +{'foo': 'bar', 'driver': 'mysql', 'user': 'alau'} +``` From 3d188cb74402e8cd7d64eeab055cc994e3e1036a Mon Sep 17 00:00:00 2001 From: Jieru Hu Date: Thu, 10 Dec 2020 15:12:17 -0800 Subject: [PATCH 3/3] address comments --- website/docs/fb/configerator-config-source.md | 25 +++++---- .../fb/configerator-config-source.md | 51 ------------------- 2 files changed, 16 insertions(+), 60 deletions(-) delete mode 100644 website/versioned_docs/version-1.0/fb/configerator-config-source.md diff --git a/website/docs/fb/configerator-config-source.md b/website/docs/fb/configerator-config-source.md index 63a1f3d851f..65589ca335d 100644 --- a/website/docs/fb/configerator-config-source.md +++ b/website/docs/fb/configerator-config-source.md @@ -14,15 +14,22 @@ Add the following to your `TARGET` file ### Usage -1. Since `configerator` does not have a client for listing configs under a path, you need to group your configs under -a `domain` for Hydra to access. - 1. Matching the name of your domain and directory of configs is necessary for the plugin to extract information on the full config names, due to limitations from Configerator APIs - 1. For example, the config paths returned by Configerator API could look like `fair_infra/hydra_plugins/configerator_config_source/example/db/mysql`. The plugin needs to know where the directory of configs begins ([`example`](https://fburl.com/diffusion/7c0c5tig)), in order to determine the full config name (`db/mysql`). So in this case the domain should be named [`example.cconf`](https://fburl.com/diffusion/pyymoo1t) -1. Create a [SearchPathPlugin](https://hydra.cc/docs/next/advanced/search_path) to add the Configerator path to the list of search paths. - - 1. The path you add in your SearchPathPlugin should be the name of your domain of configs, such as in this [example SearchPathPlugin](https://fburl.com/diffusion/ljggtux5) - - Note: There will be an easier way to add the Configerator path to the list of search paths, with a planned change in Hydra 1.1 to [allow configuring the search path via the config file](https://github.com/facebookresearch/hydra/issues/274). +1. The Configerator Config Source plugin requires that you place the configs under a [domain](https://fburl.com/wiki/n5cgchxe). +You can find an example domain [here](https://fburl.com/diffusion/ms50g5hu) + +:::important +Due to the limitations of Configerator APIs, matching the name of your domain and directory of configs is necessary for the plugin to extract information on the full config names. +For example, the config paths returned by Configerator API could look like `fair_infra/hydra_plugins/configerator_config_source/example/db/mysql`. The plugin needs to know where the directory of configs begins ([`example`](https://fburl.com/diffusion/7c0c5tig)), in order to determine the full config name (`db/mysql`). So in this case the domain should be named [`example.cconf`](https://fburl.com/diffusion/pyymoo1t) +::: + + +2. Create a [SearchPathPlugin](https://hydra.cc/docs/next/advanced/search_path) to add the Configerator path to the list of search paths. + The path you add in your SearchPathPlugin should be the name of your domain of configs, such as in this [example SearchPathPlugin](https://fburl.com/diffusion/ljggtux5) + +:::info +Adding a new search path will become much easier once [#274](https://github.com/facebookresearch/hydra/issues/274) is resolved, which is planned for Hydra 1.1. +::: + ### Example: diff --git a/website/versioned_docs/version-1.0/fb/configerator-config-source.md b/website/versioned_docs/version-1.0/fb/configerator-config-source.md deleted file mode 100644 index 29ceda868e2..00000000000 --- a/website/versioned_docs/version-1.0/fb/configerator-config-source.md +++ /dev/null @@ -1,51 +0,0 @@ ---- -id: fbcode-configerator-config-source -title: Configerator Config Source Plugin ---- - -The ConfigeratorConfigSource plugin makes it possible for Hydra applications to consume a domain of configs from configerator. - -### Dependency - -Add the following to your `TARGET` file -```commandline -//fair_infra/fbcode_hydra_plugins/configerator_config_source:configerator_config_source -``` - -### Usage - -1. Since `configerator` does not have a client for listing configs under a path, you need to group your configs under -a `domain` for Hydra to access. - 1. Matching the name of your domain and directory of configs is necessary for the plugin to extract information on the full config names, due to limitations from Configerator APIs - 1. For example, the config paths returned by Configerator API could look like `fair_infra/hydra_plugins/configerator_config_source/example/db/mysql`. The plugin needs to know where the directory of configs begins ([`example`](https://fburl.com/diffusion/7c0c5tig)), in order to determine the full config name (`db/mysql`). So in this case the domain should be named [`example.cconf`](https://fburl.com/diffusion/pyymoo1t) -1. Create a [SearchPathPlugin](https://hydra.cc/docs/next/advanced/search_path) to add the Configerator path to the list of search paths. - 1.The path you add in your SearchPathPlugin should be the name of your domain of configs, such as in this [example SearchPathPlugin](https://fburl.com/diffusion/ljggtux5) - - Note: There will be an easier way to add the Configerator path to the list of search paths, with a planned change in Hydra 1.1 to [allow configuring the search path via the config file](https://github.com/facebookresearch/hydra/issues/274). - -### Example: - -#### Example SearchPathPlugin -[`ConfigeratorExampleSearchPathPlugin`](https://fburl.com/diffusion/vwa82fbg) adds the example configerator domain to the search path of the example applications. - -#### Reading primary config from configerator -This example reads its primary config from configerator [here](https://fburl.com/diffusion/twk3smkj) which has a default list defined. - -```commandline -$ buck run //fair_infra/fbcode_hydra_plugins/configerator_config_source/example/primary_config:my_app -... -Parsing buck files: finished in 1.1 sec -... -{'driver': 'mysql', 'user': 'alau'} -``` - -#### Compose config with configerator -This example reads its primary config from local yaml file `primary_config.yaml` but reads config groups info from configerator. - -```commandline -$ buck run //fair_infra/fbcode_hydra_plugins/configerator_config_source/example/config_group:my_app -- +db=mysql -... -Parsing buck files: finished in 1.1 sec -... -{'foo': 'bar', 'driver': 'mysql', 'user': 'alau'} -```