feat(dfn): add dfn container, parser, toml conversion script (#167)

Add some input definition representations, a DFN file parser, and a TOML conversion utility script. These could be used from MODFLOW 6 and from FloPy until the TOML migration is complete.

Also add an optional dependency group 'dfn' which includes boltons and tomlkit.

.github/workflows/ci.yml

@@ -57,8 +57,7 @@ jobs:
       - name: Install Python packages
         run: |
           pip install --upgrade pip
-          pip install build twine
-          pip --verbose install .
+          pip install ".[build]"
 
       - name: Print package version
         run: python -c "import modflow_devtools; print(modflow_devtools.__version__)"
@@ -134,7 +133,7 @@ jobs:
 
       - name: Install Python packages
         working-directory: modflow-devtools
-        run: pip install ".[test]"
+        run: pip install ".[dev]"
 
       - name: Cache modflow6 examples
         id: cache-examples

.gitignore

@@ -143,4 +143,6 @@ app
 bin
 
 **.DS_Store
-data_backup
+data_backup
+
+autotest/temp/

autotest/test_dfn.py

@@ -0,0 +1,33 @@
+from pathlib import Path
+
+from modflow_devtools.dfn import Dfn, get_dfns
+from modflow_devtools.markers import requires_pkg
+
+PROJ_ROOT = Path(__file__).parents[1]
+DFN_PATH = PROJ_ROOT / "autotest" / "temp" / "dfn"
+MF6_OWNER = "MODFLOW-USGS"
+MF6_REPO = "modflow6"
+MF6_REF = "develop"
+
+
+def pytest_generate_tests(metafunc):
+    if "dfn_name" in metafunc.fixturenames:
+        if not any(DFN_PATH.glob("*.dfn")):
+            get_dfns(MF6_OWNER, MF6_REPO, MF6_REF, DFN_PATH, verbose=True)
+        dfn_names = [
+            dfn.stem
+            for dfn in DFN_PATH.glob("*.dfn")
+            if dfn.stem not in ["common", "flopy"]
+        ]
+        metafunc.parametrize("dfn_name", dfn_names, ids=dfn_names)
+
+
+@requires_pkg("boltons")
+def test_dfn_load(dfn_name):
+    with (
+        (DFN_PATH / "common.dfn").open() as common_file,
+        (DFN_PATH / f"{dfn_name}.dfn").open() as dfn_file,
+    ):
+        common, _ = Dfn._load_v1_flat(common_file)
+        dfn = Dfn.load(dfn_file, name=dfn_name, common=common)
+        assert any(dfn)

docs/md/dfn.md

@@ -0,0 +1,27 @@
+# Working with definition files
+
+MODFLOW 6 specifies input components and their variables with a custom file format. Input specification files are called definition (DFN) files and conventionally have suffix `.dfn`.
+
+Work is underway to migrate MODFLOW 6 input specifications to a standard data interchange format. TOML has tentatively been selected.
+
+The `modflow_devtools.dfn` module contains a parser for the legacy DFN format, a format-agnostic representation for input specifications, and a TOML conversion utility.
+
+We envision MODFLOW 6 and FloPy will use these utilities for a relatively short period while the TOML migration is underway. This will involve adapting automated code- and documentation-generation systems in both MF6 and FloPy to consume TOML rather than DFN files. When this is complete, these utilities should no longer be necessary.
+
+## TOML conversion
+
+The `dfn` optional dependency group is necessary to use the TOML conversion utility.
+
+To convert definition files to TOML, use:
+
+```shell
+python -m modflow_devtools.dfn.dfn2toml -i <path to dfns> -o <output dir path>
+```
+
+### Format
+
+The TOML format is structurally different from the original DFN format: where legacy DFNs are flat lists of variables, with comments demarcating blocks, a TOML input definition is a tree of blocks, each of which contains child variables, each of which can be a scalar or a composite &mdash; composites contain their own child variables. The definition may also contain other top-level attributes besides blocks, so long as they do not conflict with block names.
+
+Children are not explicitly marked as such &mdash; rather children are attached directly to the parent and can be identified by their type (i.e., as a dictionary rather than a scalar).
+
+While structurally different, TOML definition files are visually similar to the DFN format, due to the fact that TOML represents hierarchy with headers, rather than indentation as in YAML.