refactor: simplify autotest framework #1464

wpbonelli · 2023-11-23T05:22:52Z

Changes include

fold TestSimulation into TestFramework
rename/refactor properties
- simpath -> workspace
- build_function -> build (and inject in __init__)
- exfunc -> check
- exe_dict -> targets
- require_failure -> xfail
- action -> compare
- fold switches run_comparison, make_comparison, mf6_regression into compare (optional str), accepts:
  - "auto" picks comparison type based on workspace contents, equivalent to previous "compare" (default)
  - "run_only" equivalent to previous run_comparison=True and make_comparison=False
  - "mf6_regression" equivalent to previous mf6_regression=True
  - None disables comparison
standardize test function names, step toward automatic discovery?
- build_models creates models
- check_output checks results
framework doesn't need to know about test case indices, remove idxsim and just use lambda if build_models() needs a test case index
support pathlib.Path or str paths
run black/isort, format test description docstrings
remove unused functions from common_regression.py
remove pytest-cases from tests and as a test dependency
fix hardcoded gridgen left in conftest.py from test: require dev exes, warn/skip if download/rebuilt not found #1460

Merge considerations

requires modflow-devtools >= 1.1 (update environment.yml to specify this)
should probably wait for fix: update SFR diversion flow to zero when its upstream downflow is zero #1467 and any other open PRs touching autotests, so any conflicts can be resolved here

Future directions

support multiple comparison/regression models? (e.g. compare current mf6 with previous mf6 and mf2005, or PRT with mp6 and mp7)
explicit bookkeeping for sim/model workspaces? (replace implicit expectation that test and framework agree)
autodiscovery of test cases following e.g. this pattern?

jdhughes-usgs · 2023-12-04T15:03:42Z

autotest/common_regression.py

@@ -160,15 +161,15 @@ def setup_comparison(namefile, dst, remove_existing=True):
                print("Could not make " + dst)
        # clean directory
        else:
-            print(f"cleaning...{dst}")
+            print("Cleaning ", dst)


The change is fine but I am wondering if there is a reason to move away from f-strings

updated to use f-strings consistently and wrap paths with single quotes e.g. Cleaning 'some/folder' as it's a bit easier to read

jdhughes-usgs · 2023-12-04T15:04:56Z

autotest/common_regression.py

    # Possible comparison - the order matters
-    optcomp = (
-        "compare",
+    OPTCOMP = (


I like the change to uppercase for defined data.

I meant to move this to a top-level variable so other modules could import it, will do so if you agree

edit: done and renamed to COMPARE_PATTERNS

autotest/common_regression.py

* fold TestSimulation into TestFramework class * rename simpath -> workspace * rename build_func -> build * rename exfunc -> check * rename exe_dict -> targets * support pathlib.Path throughout * steps toward unifying framework function signature

…ses to fn params

…avor of enumeration for simpler external API, still kluged internally

…pted with MODFLOW-USGS#1464

* feat(GWE): Introduce Groundwater Energy Transport * fix typo in meson file * fix a formating issue that was popping up in an MST autotest * some legacy line leftover from a botched rebase, possibly * Need update initial autotest to conform to new autotest standards adopted with #1464 * Start looking for post-rebase breakages * Get gwfgwe and gwegwe exchanges up-to-date based on #1505 * Code that had been moved to set_active_status in FMI was still present in fmi_fc(). Removing. * Forgot to remove unused variables after making changes in 4d76729 * Adding another autotest specific to GWE * Adding another autotest after getting it #1464 compliant * Compliantizing another new autotest with PR #1464 * Fixes in response to #1493 (comment) * Fix in response to #1493 (comment) * Rerunning black in response to #1493 (comment) * Fix in response to #1493 (comment) * Made changes to dfn files in response to #1493 (comment) and #1493 (comment) and reran mf6ivar.py, which updated the tex files in this commit * Fix in response to #1493 (comment) * remove unnecessary line of script * Add missing lines to ConnectionBuilder.f90 related to GWE * Add a GWE vs GWT comparison autotest * GWE-GWE exchanges now working. Other clean-up for code uniformity * Rebrand constant temperature package acronym to CTP * Missed an import renaming update * Forgot to remove a now obsolete file due to renaming. * Rebrand energy storage and transfer package acronym to EST * Forgot to remove a now obsolete file due to renaming (again) * Rebrand gwe dispersion package acronym to CND (conduction) since that is the dominant process in heat transport * Remove gwe-related code from generalized transport code (tsp1.f90 & tsp1ssm1.f90) * add single-cell test for energy source loading (ESL) package * Adding energy source loading (ESL) package * remove unused variable * fprettify * Add another ESL autotest * Adding another autotest that compares gwe to three different analytical solutions from Carslaw & Jaeger (1947) * Bringing over Stallman autotest from previous GWE PR (#1237) * Adding streamflow energy transport (SFE) package * Add autotest for SFE * Adding lake energy transport (LKE) package. Includes new autotest * forgot meson update * Adding multi-aquifer well energy transport (MWE) package. Includes new autotest * Adding unsaturated-zone energy transport (UZE) package. Includes 2 new autotests * Attempting to reapply a failing autotest. Unable to discern why it is failing as downloaded contents from its failure are working locally. * removing troublesome autotest. Downloaded contents from failed run on Actions doesn't yield the same output that the logs are reporting. This particular autotest has an analytical solution (and a plot) that might be better shown on the modflow6-examples repo anyway * Removing a file that shouldn't have been added (snuck in among other staged files) * Update release notes

wpbonelli marked this pull request as ready for review November 24, 2023 20:20

wpbonelli force-pushed the autotests branch from 9aa0784 to 74230df Compare November 30, 2023 18:50

wpbonelli requested a review from jdhughes-usgs December 4, 2023 14:57

jdhughes-usgs approved these changes Dec 4, 2023

View reviewed changes

emorway-usgs mentioned this pull request Dec 4, 2023

fix: update SFR diversion flow to zero when its upstream downflow is zero #1467

Merged

wpbonelli added 25 commits December 5, 2023 10:30

refactor: simplify autotest framework

d2c4025

* fold TestSimulation into TestFramework class * rename simpath -> workspace * rename build_func -> build * rename exfunc -> check * rename exe_dict -> targets * support pathlib.Path throughout * steps toward unifying framework function signature

fix path printing in common_regression.py

507fb27

step closer to unified hooks, formatting

58f593c

fix failing tests, formatting

f4ea10b

remove separate libmf6 fixture (just use targets)

44152f2

slight refactor to get_mf6_files()

5032bd7

minor framework cleanup, docstrings in common_regression.py

8e45a3e

move external model tests to autotest/example

d922a86

simplify comparison fns

ad69f8d

misc refactoring, enforce toggles in top-level run(), lift prop acces…

da31cec

…ses to fn params

action -> comparison, fold mf6_regression into comparison option

8d161fc

misc fixes

3e6c3bc

fix _compare_heads()

f6f3185

cleanup in example/* tests

1b78b2b

fix testmodel/largetestmodel tests

4419e60

remove pytest-cases usages and dependency

c9ce519

rename comparison -> compare, remove comparison-related switches in f…

59e8f0e

…avor of enumeration for simpler external API, still kluged internally

remove nested autotest/example dir, minor cleanup

7f853eb

more cleanup, print statement formatting

aee7f5b

test fixes

fa3dd18

switch to auto as default option, more cleanup

8eea50e

prep to support multiple comparison sims

fab92a6

fix DEVELOPER.md

afa70fe

require modflow-devtools>=1.1

2c22a38

fix GWT -> GWT6 model key

743da91

emorway-usgs added a commit to geohackweek/modflow6-tempfork that referenced this pull request Jan 2, 2024

Adding another autotest after getting it MODFLOW-USGS#1464 compliant

afb4ba0

emorway-usgs added a commit to geohackweek/modflow6-tempfork that referenced this pull request Jan 2, 2024

Compliantizing another new autotest with PR MODFLOW-USGS#1464

a422795

emorway-usgs added a commit to geohackweek/modflow6-tempfork that referenced this pull request Jan 17, 2024

Need update initial autotest to conform to new autotest standards ado…

ccd428d

…pted with MODFLOW-USGS#1464

emorway-usgs added a commit to geohackweek/modflow6-tempfork that referenced this pull request Jan 17, 2024

Adding another autotest after getting it MODFLOW-USGS#1464 compliant

4e8ef44

emorway-usgs added a commit to geohackweek/modflow6-tempfork that referenced this pull request Jan 17, 2024

Compliantizing another new autotest with PR MODFLOW-USGS#1464

2723904

emorway-usgs added a commit to geohackweek/modflow6-tempfork that referenced this pull request Jan 19, 2024

Need update initial autotest to conform to new autotest standards ado…

6a38f11

…pted with MODFLOW-USGS#1464

emorway-usgs added a commit to geohackweek/modflow6-tempfork that referenced this pull request Jan 19, 2024

Adding another autotest after getting it MODFLOW-USGS#1464 compliant

7ebf656

emorway-usgs added a commit to geohackweek/modflow6-tempfork that referenced this pull request Jan 19, 2024

Compliantizing another new autotest with PR MODFLOW-USGS#1464

0e4ce64

emorway-usgs added a commit to geohackweek/modflow6-tempfork that referenced this pull request Jan 19, 2024

Need update initial autotest to conform to new autotest standards ado…

81ddb1f

…pted with MODFLOW-USGS#1464

emorway-usgs added a commit to geohackweek/modflow6-tempfork that referenced this pull request Jan 19, 2024

Adding another autotest after getting it MODFLOW-USGS#1464 compliant

415e232

emorway-usgs added a commit to geohackweek/modflow6-tempfork that referenced this pull request Jan 19, 2024

Compliantizing another new autotest with PR MODFLOW-USGS#1464

c5446e1

emorway-usgs added a commit to geohackweek/modflow6-tempfork that referenced this pull request Jan 26, 2024

Need update initial autotest to conform to new autotest standards ado…

a68ab0f

…pted with MODFLOW-USGS#1464

emorway-usgs added a commit to geohackweek/modflow6-tempfork that referenced this pull request Jan 26, 2024

Adding another autotest after getting it MODFLOW-USGS#1464 compliant

f9d7509

emorway-usgs added a commit to geohackweek/modflow6-tempfork that referenced this pull request Jan 26, 2024

Compliantizing another new autotest with PR MODFLOW-USGS#1464

bb30533

emorway-usgs added a commit to geohackweek/modflow6-tempfork that referenced this pull request Feb 6, 2024

Need update initial autotest to conform to new autotest standards ado…

17878b3

…pted with MODFLOW-USGS#1464

emorway-usgs added a commit to geohackweek/modflow6-tempfork that referenced this pull request Feb 6, 2024

Adding another autotest after getting it MODFLOW-USGS#1464 compliant

1391987

emorway-usgs added a commit to geohackweek/modflow6-tempfork that referenced this pull request Feb 6, 2024

Compliantizing another new autotest with PR MODFLOW-USGS#1464

fd7b3a3

emorway-usgs added a commit to geohackweek/modflow6-tempfork that referenced this pull request Feb 8, 2024

Need update initial autotest to conform to new autotest standards ado…

e65199e

…pted with MODFLOW-USGS#1464

emorway-usgs added a commit to geohackweek/modflow6-tempfork that referenced this pull request Feb 8, 2024

Adding another autotest after getting it MODFLOW-USGS#1464 compliant

255ded6

emorway-usgs added a commit to geohackweek/modflow6-tempfork that referenced this pull request Feb 8, 2024

Compliantizing another new autotest with PR MODFLOW-USGS#1464

2bffa06

wpbonelli pushed a commit to wpbonelli/modflow6 that referenced this pull request Feb 9, 2024

Need update initial autotest to conform to new autotest standards ado…

f834f5b

…pted with MODFLOW-USGS#1464

wpbonelli pushed a commit to wpbonelli/modflow6 that referenced this pull request Feb 9, 2024

Adding another autotest after getting it MODFLOW-USGS#1464 compliant

184b599

wpbonelli pushed a commit to wpbonelli/modflow6 that referenced this pull request Feb 9, 2024

Compliantizing another new autotest with PR MODFLOW-USGS#1464

0f74c81

emorway-usgs added a commit to geohackweek/modflow6-tempfork that referenced this pull request Feb 9, 2024

Need update initial autotest to conform to new autotest standards ado…

060e2f9

…pted with MODFLOW-USGS#1464

emorway-usgs added a commit to geohackweek/modflow6-tempfork that referenced this pull request Feb 9, 2024

Adding another autotest after getting it MODFLOW-USGS#1464 compliant

f27fd94

emorway-usgs added a commit to geohackweek/modflow6-tempfork that referenced this pull request Feb 9, 2024

Compliantizing another new autotest with PR MODFLOW-USGS#1464

147dd7a

emorway-usgs added a commit to geohackweek/modflow6-tempfork that referenced this pull request Feb 9, 2024

Need update initial autotest to conform to new autotest standards ado…

00a6e82

…pted with MODFLOW-USGS#1464

emorway-usgs added a commit to geohackweek/modflow6-tempfork that referenced this pull request Feb 9, 2024

Adding another autotest after getting it MODFLOW-USGS#1464 compliant

b24a919

emorway-usgs added a commit to geohackweek/modflow6-tempfork that referenced this pull request Feb 9, 2024

Compliantizing another new autotest with PR MODFLOW-USGS#1464

b528234

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: simplify autotest framework #1464

refactor: simplify autotest framework #1464

wpbonelli commented Nov 23, 2023 •

edited

Loading

jdhughes-usgs Dec 4, 2023

wpbonelli Dec 4, 2023

jdhughes-usgs Dec 4, 2023

wpbonelli Dec 4, 2023 •

edited

Loading

refactor: simplify autotest framework #1464

refactor: simplify autotest framework #1464

Conversation

wpbonelli commented Nov 23, 2023 • edited Loading

jdhughes-usgs Dec 4, 2023

Choose a reason for hiding this comment

wpbonelli Dec 4, 2023

Choose a reason for hiding this comment

jdhughes-usgs Dec 4, 2023

Choose a reason for hiding this comment

wpbonelli Dec 4, 2023 • edited Loading

Choose a reason for hiding this comment

wpbonelli commented Nov 23, 2023 •

edited

Loading

wpbonelli Dec 4, 2023 •

edited

Loading