stage 1: kedro parameters <-> dvc parameters

[shaunc]

Representation

Kedro parameters are configured project config/base/config/local and are located relative to there or to project root (searched in order local, base, then project root). Parameters are stored in yaml files.

Kedro allows parameters to be consumed directly in python code. Or they can be consumed as node inputs. Credentials are managed separately, and can be used in the data catalog. Secure credentials are intended to be placed in config/local/credentials (config/local should be in .gitignore).

Kedro parameters are by default inside the configuration environment (typically in conf/base or conf/local). They are referenced in inputs using the syntax params:<key> to reference individual parameters. (Unlike DVC, it would seem that the key is not a dottedname, but references a top-level entity in parameters.

DVC supports a wide range of formats for parameter specification. They are referenced in pipeline files. Credentials are not tracked, as reading and writing of files is not modelled by dvc.

DVC specifies parameters in two different manners -- in params section in each stage, and via templates vars section, which can be ether global to pipeline, or local to stage.

DVC has a default params file (params.yaml, but other formats are also supported). The stage params section is a list of parameters, keys may be specified directly, or they may be specified as a dictionary entry:

stage:
  params:
  - <key>
  - <parameter file name>:
    - <key>

In either case, <key> is a dotted name referencing a possibly nested value in the parameter file.

In dvc.yaml templates, parameters can be used not only as inputs to stage execution, but also can vary other stage metadata.

Implementation

Only parameters used as node inputs can be tied to experiments. Kedro-DVC will include parameters referenced by nodes using the params:<key> syntax in dvc.yaml stage definitions.

Prev	Up	Next
data	stage 1	artifacts

[antonymilne]

Just a few things to add here (some I already mentioned on discord, but let me put them here just for completeness).

1. Under the hood, kedro's ConfigLoader uses anyconfig. This actually supports a whole range of of configuration formats (listed in our documentation is yaml, json, ini, pickle, xml, properties, but maybe anyconfig does even more now). With that said, yaml is overwhelmingly the most popular option (no exaggeration when I say that I'd guess that >99.99% of kedro config files are yaml)
2. Referencing subkeys of parameters is possible and looks quite similar to DVC: params:key.subkey.subsubkey
3. "Kedro allows parameters to be consumed directly in python code. Or they can be consumed as node inputs." While there's nothing technically stopping you from manually loading up a parameters file and consuming it in your node code, this is very strongly discouraged and shouldn't ever really happen. So "Only parameters used as node inputs can be tied to experiment" is a very reasonable limitation
4. ConfigLoader is something which does have a well-defined interface, and kedro ships with two implementations of that. TemplatedConfigLoader enables templating through injection of global variables and Jinja2
5. ConfigLoader is probably the easiest and most common kedro component to swap out in settings.py, e.g. people sometimes want to change the run environment hierarchy from local > base to something more complex like local > env-subenv > env > base. Not sure if a custom config loader will be any use on kedro-dvc, but just to say that if it is required then it should be relatively easy.

In general the topic of configuration (usually in relation to catalog entries rather than parameters) is quite a hot topic in kedro and there's lots of different opinions on it, e.g. kedro-org/kedro#1076, kedro-org/kedro#891, kedro-org/kedro#770, kedro-org/kedro#1011. So chances are if you need to do something a bit out of the ordinary here, someone has considered how to do it already at some point.

[antonymilne]

Unfortunately I don't think there is. The only thing that's present in catalog.datasets are the names of the datasets, which will include all the possible params:* patterns that would be valid node inputs. The information about which files these were originally defined in has long gone by that point. In fact this information is never really available within kedro. This is how parameters are loaded in the first place. If you follow that does does through to lower level details then ultimately the best you can easily achieve is to get a list of all parameters file that kedro looks at, as we do here. There's never really a mapping of which parameters are defined in which files available anywhere.

[shaunc]

Hmm... and then _path_lookup is called by _lookup_config_filepaths which sorts the (arbitrarily ordered) files, and that order determines how parameters are resolved. Then that in turn is called by _get_config_from_patterns which throws out the filepaths. This is finally called by ConfigLoader and/or TemplatedConfigLoader.

Sigh... so it would seem that we'll have to create this logic over again.

Any chance you'd accept a PR to bubble up a map from key -> filepath and use it in specializations of AbstractConfigLoader? (An instance of which would be available in the context?)

Aside -- it would seem that would be nice if this were memoizing? (Or isn't it called very much?)

    def get(self, *patterns: str) -> Dict[str, Any]:
        return _get_config_from_patterns(
            conf_paths=self.conf_paths, patterns=list(patterns)
        )

[antonymilne]

Very reasonable though that sounds, to be honest I think such a PR is unlikely to be accepted, because the whole topic of config in kedro is such a big one that we're trying to deal with it holistically and not change just small pieces of it any more. Sorry! Note that even if the key -> filepath map were available after ConfigLoader you would still need to make it available in catalog somehow. Unfortunately I think you'll have to come up with some workaround here:

• recreate the logic yourself
• or write a new ConfigLoader (and also DataCatalog) which do what you need them to
• link the dependency of a parameter input to all available parameter files rather than just the source one

[antonymilne]

N.B. even though the last suggestion sounds very crude, it's worth noting that even if you did have the key -> filepath map available this still isn't ideal. e.g. if my node input is params:key.subkey then what I really care about is whether the hash of that particular value has changed, not the hash of the file it's contained in, which could contain any number of irrelevant parameters that I'm changing. A key -> filepath map would only be sufficient if you break your parameters files up so that each node's parameter dependencies is stored in a separate file (which might not be a bad way to work but definitely isn't always the case in kedro).

[shaunc]

A key -> filepath map would only be sufficient if you break your parameters files up...

My thought here was to use "file" dependencies for dvc, then (in stage 2, perhaps) do fine-grained skip based on particular parameter. (But how would I know what the old value is? Am I caching them somewhere?)

HOWEVER -- I just may have hit on a way to avoid this who morass -- tell me what you think!

What if I just write new parameter files in conf/kedro_dvc/ and point dvc at them? When run starts, before I ask DVC which nodes to run, I can write in the most recent parameters.

I can have one file per parameter. (Perhaps this could be configurable someday....)