Code

Feature 1546 CI testing#2029

Code

Merged

JohnHalleyGotway merged 147 commits intodevelopdtcenter/MET:developfrom feature_1546_unit_xmlsdtcenter/MET:feature_1546_unit_xmlsCopy head branch name to clipboardJan 27, 2022

+3,621-2,274

Commits

Commits on Jan 18, 2022

• use env var to set MET_PYTHON_EXE in unit tests instead of using hard-coded path

  georgemccabe

  committed
  7461330

• ignore log file created in Docker image

  georgemccabe

  committed
  ca71c58

• rename output file from test to no longer contain + character that causes issues with NCO utilities

  georgemccabe

  committed
  760afd0

• wrote script that contains all commands that are needed to install what is needed to run MET unit tests and diff logic in Docker

  georgemccabe

  committed
  a06a71f

Commits on Jan 19, 2022

• split script to build and push MET docker image into 2 scripts

  georgemccabe

  committed
  e337b08

• removed echo commands to group log output to allow script/job to fail if the docker commands fail. The scripts are now called in separate steps that generate their own log files so these groups are…

  Show description for 303ab44
  georgemccabe

  committed
  303ab44

• Per #1546, move steps to install external libraries needed to install MET and tools needed to run unit tests into seperate Dockerfiles

  georgemccabe

  committed
  d3a0487

Commits on Jan 20, 2022

• per #1546, moved logic to build MET in Docker into a script to call from a Dockerfile so that it can be called from Dockerfile and Dockerfile.copy

  georgemccabe

  committed
  d07fcb0

• Per #1546, added Dockerfile that copies local MET source code into Docker image instead of cloning the repository

  georgemccabe

  committed
  5cd2afb

• Per #1546, fixed path to build script

  georgemccabe

  committed
  04c72a3

• rename file to match DockerHub image name

  georgemccabe

  committed
  d787dbb

• Per #1546, set env var MET_REPO_DIR to easily get directory that repository was cloned into

  georgemccabe

  committed
  f9b2a19

• Per #1546, changed script to run unit tests and comparison, writing to log files

  georgemccabe

  committed
  2a1579c

• Per #1546, check return code of commands and exit non-zero if failure. This was not needed in Dockerfile because the commands were chained together with &&, but in a script execution will continue …

  Show description for 0ab6fd7
  georgemccabe

  committed
  0ab6fd7

• added readme file to describe how to use each Dockerfile in the directory

  georgemccabe

  committed
  c0d233f

• Per #1546, added error checking and reporting to unit test script

  georgemccabe

  committed
  e85c79c

• Per #1546, moved time_command function into a file to be sourced by each script that needs to use it

  georgemccabe

  committed
  f2dbb7d

• Per #1546, changed build docker script to use Dockerfile.copy to use local source code, set environment variables that set docker build args in workflow yml

  georgemccabe

  committed
  3c55389

• Per #1546, renamed jobs and steps to be more clear

  georgemccabe

  committed
  6de4ac5

• Per #1546, fixed logic to get branch name

  georgemccabe

  committed
  0b70c80

• Per #1546, fixed logic to get branch name in other workflow

  georgemccabe

  committed
  d19ccfa

• changed event rules to trigger on push to feature and bugfix branches on changes under met/src only

  georgemccabe

  committed
  06c4527

• Per #1546, to test automation logic, make change to file in met/src that should trigger compilation test and make change that should break compilation

  georgemccabe

  committed
  0327436

• per #1546, pipe docker build output to log file and make log available as artifact on failure

  georgemccabe

  committed
  2bdefad

• Per #1546, testing: another change under met/src to trigger compilation test

  georgemccabe

  committed
  ed8e212

• fix path to file to upload as artifact

  georgemccabe

  committed
  f1525cc

• test again

  georgemccabe

  committed
  e511cdc

• fix sending output to logfile while keeping timing information in stdout

  georgemccabe

  committed
  ac13a16

• another change to test broken compilation

  georgemccabe

  committed
  5dc9419

• remove line that causes error

  georgemccabe

  committed
  21f6505

• last test for compilation failure

  georgemccabe

  committed
  78bb8c7

• fixed error catching when log file is set

  georgemccabe

  committed
  c4ef297

• test failed compilation one more time

  georgemccabe

  committed
  07bd039

• formatting

  georgemccabe

  committed
  0679504

• always upload build log artifact so it can be easily reviewed on a successful build of MET

  georgemccabe

  committed
  3b75672

• fixed intentional bug used for testing

  georgemccabe

  committed
  aadefd8

• minor change to trigger workflow on push

  georgemccabe

  committed
  98327f4

• Per #1546, add option to skip MET compilation test by setting ci-skip-compile

  georgemccabe

  committed
  40dcd22

• trigger compilation test

  georgemccabe

  committed
  fc442b7

• fixed logic to skip compile test by setting ci-skip-compile

  georgemccabe

  committed
  ba2ed67

• always run upload artifact step regardless of failure or success

  georgemccabe

  committed
  5721ece

• test once again

  georgemccabe

  committed
  c97258c

• change code back to original state

  georgemccabe

  committed
  de40002

• added workflow to run unit tests, compare output to truth data, and create artifacts with log files and output files that differ between runs

  georgemccabe

  committed
  a390ca1

• changed name of job, fixed typo

  georgemccabe

  committed
  b8c6099

• fixed command run inside docker

  georgemccabe

  committed
  286c00d

• pass GITHUB_WORKSPACE env var into docker container

  georgemccabe

  committed
  0cd00a5

• fixed reference to runner workspace from workflow file

  georgemccabe

  committed
  16a1ae8

• remove GITHUB_WORKSPACE from docker run command and refer to script relative to env var set in docker image

  georgemccabe

  committed
  455dab0

• change again to trigger compile test to check that changes work as expected

  georgemccabe

  committed
  0383320

• run step to copy build log to log directory even if build fails

  georgemccabe

  committed
  d6fe01b

• changed code back to original state after confirmed behavior in automation

  georgemccabe

  committed
  f090976

• Per #1546, added function to get branch name depending on if event is pull request or push. Set unit test workflow to trigger on pull requests into develop branch

  georgemccabe

  committed
  41d4040

• set MET_FONT_DIR with ENV so it is avaiable for test runs

  georgemccabe

  committed
  192b0d3

Commits on Jan 21, 2022

• catch error if copy script fails

  georgemccabe

  committed
  c554789

• fix typo in shebang line

  georgemccabe

  committed
  e3ec594

• Per #1546, fixed incorrect directory to mount diff output

  georgemccabe

  committed
  7a592a9

• fix bad path that causes diff artifact upload to fail

  georgemccabe

  committed
  6def40f

• Per #1546, many changes including adding new job control rules, building docker image and pushing to dockerhub in a job then pulling image in other jobs, run subset of unit tests defined by a matrix

  georgemccabe

  committed
  f4a15ca

• fix typo in script

  georgemccabe

  committed
  979de4b

• don't run unit tests unless job control specifies to

  georgemccabe

  committed
  ca246ba

• echo job control settings so they can be seen in the logs, trigger unit tests with ci-run-unit

  georgemccabe

  committed
  346517e

• added always() to unit test if so that it will still run even if compile step is skipped

  georgemccabe

  committed
  71fc960

• added escaped quotes around tests argument to prevent failure, ci-skip-compile

  georgemccabe

  committed
  7c6552f

• unit tests do not run if compile is skipped, so removing always() from if

  georgemccabe

  committed
  ce2a635

• fix check for empty diff dir

  georgemccabe

  committed
  f060c0a

• temporarily remove unit test dependency on compile for testing, ci-skip-compile, ci-run-unit

  georgemccabe

  committed
  ec44bb9

• pass in list of tests via env var instead of argument, ci-skip-compile, ci-run-unit

  georgemccabe

  committed
  df1d3e0

• try exporting env var so it is passed into docker container, ci-skip-compile, ci-run-unit

  georgemccabe

  committed
  dd476b9

• testing to debug issues in automation, ci-skip-compile, ci-run-unit

  georgemccabe

  committed
  9d38763

• need to compile to get changes, ci-run-unit

  georgemccabe

  committed
  9a4f7da

• fixed typo in var name, ci-run-unit

  georgemccabe

  committed
  875a90e

• split unit test runs and diff logic into separate steps so it is easier to see where a failure occurs, ci-run-unit

  georgemccabe

  committed
  9ec546c

Commits on Jan 22, 2022

• try mounting all volumes in unit test docker run, then docker exec to run the diff tests, ci-run-unit

  georgemccabe

  committed
  3d1b8ce

Commits on Jan 24, 2022

• start docker container then run separate commands in container, ci-run-unit

  georgemccabe

  committed
  94f7283

• create container with ID to use to run commands, ci-run-unit

  georgemccabe

  committed
  96c62b3

• try running docker with --name, ci-run-unit

  georgemccabe

  committed
  c1ddd15

• try starting container then using docker exec to run commands inside, ci-run-unit

  georgemccabe

  committed
  ec6896c

• try running docker with -d (detach) to run in background with volumes mounted and run docker exec commands to execute scripts, ci-run-unit

  georgemccabe

  committed
  a14b920

• temporary skip compile to test running docker, ci-skip-compile, ci-run-unit

  georgemccabe

  committed
  4574861

• append to logfile if it already exists

  georgemccabe

  committed
  ffadbe8

• add first test in group to log file name for copying diff files

  georgemccabe

  committed
  1e0aa8a

• consolidate functions calls into fewer steps, copy unit test job into multiple with varying dependencies on other test groups, ci-run-unit

  georgemccabe

  committed
  ee76465

• instead of failing if a diff is found in a unit test, run a job at the end that checks if any diff files were found and fails if so, ci-run-unit

  georgemccabe

  committed
  4f46e49

• fixed name of dependent jobs, remove checkout action since we don't need source code for this job

  georgemccabe

  committed
  ada5b32

• set unit test groups based on dependencies and length of run, ci-run-unit

  georgemccabe

  committed
  3504df9

Commits on Jan 25, 2022

• fixed logic parsing missing file section to get correct full path to directories, ci-run-unit

  georgemccabe

  committed
  806bb02

• print which comp_dir log file is being parsed, ci-run-unit

  georgemccabe

  committed
  3d02253

• added _group to log file name that contains first test name to make it more clear that the file contains info for more than that test, ci-run-unit

  georgemccabe

  committed
  b321d56

• merged develop after #2021

  georgemccabe

  committed
  b79bf7e

• Per #1546, rearrange use case group dependencies, save output as artifact to download in dependent step, download all output artifacts and run diff at end of workflow

  georgemccabe

  committed
  8d74b48

• copy all output data artifact content into output directory, ci-run-unit

  georgemccabe

  committed
  dd26457

• Per #1546, change output filename to use underscore instead of colon because that is an invalid character for GHA artifacts

  georgemccabe

  committed
  f37e09c

• Per #1546, changed output filenames that include colon to use underscore instead

  georgemccabe

  committed
  24ad533

• Per #1546, update name of grib_tables unit test group from test_grib_tables since this was changed in PR #2001, ci-run-unit

  georgemccabe

  committed
  0bb89d8

• use vars to set local and docker dirs to mount, create local dirs

  georgemccabe

  committed
  0664680

• added job_control to needs list for unit test jobs so they can get the job control output variables, give unit test jobs unique names, ci-run-unit

  georgemccabe

  committed
  3a7436c

• Per #1546, remove check of content of output directory in unit tests 1a, create output directory before copying unit test artifact files for diff job, ci-run-unit

  georgemccabe

  committed
  14669d7

• added missing checkout action before running diff script, ci-run-unit

  georgemccabe

  committed
  3dc9673

• add diff logs to artifact, ci-run-unit

  georgemccabe

  committed
  0989ea3

• print contents of log file on failure for easier debugging

  georgemccabe

  committed
  fb6cd8d

• fixed variable that contains log file to parse, ci-run-unit

  georgemccabe

  committed
  3ed5015

• removed import that is no longer used

  georgemccabe

  committed
  33dbae7

• added logic to set job control variables to run diff and update truth data

  georgemccabe

  committed
  ffeffad

• added logic to set job control variables to run diff and update truth data

  georgemccabe

  committed
  6d1c585

• Per #1546, fix copy of data into dir that is mounted to Docker so diff tests will succeed, ci-run-unit

  georgemccabe

  committed
  8f783d2

• Per #1546, added logic to update truth data volume on DockerHub for -ref branches

  georgemccabe

  committed
  088bcb1

• remove dir listing for output and trigger ci-run-unit to test with data from kiowa

  georgemccabe

  committed
  2620aa4

Commits on Jan 26, 2022

• Per #1546, split output independent ascii2nc tests into unit_ascii2nc_indy.xml.

  JohnHalleyGotway

  committed
  0fc5bd5

• Per #1546, split output independent pb2nc test into unit_pb2nc_indy.xml.

  JohnHalleyGotway

  committed
  46d8785

• change event/job rules so workflow is triggered on push to feature,bugfix,develop,develop-ref,and main_v branches. Skip PR into -ref branches. Run diff logic on push for -ref branches to easily see…

  Show description for 2e37e4f
  georgemccabe

  committed
  2e37e4f

• Testing: temporarily change settings to test running all unit tests and update truth data set named develop-test

  georgemccabe

  committed
  b69407c

• test running diff logic using dtcenter/met-base:unit_test and building MET using dtcenter/met-base:minimum, ci-run-unit

  georgemccabe

  committed
  6f68825

• revert back to using full MET image for diff test since unit_test base is needed to run each unit test anyway

  georgemccabe

  committed
  e4bffe9

• revert back to using full MET image for diff test since unit_test base is needed to run each unit test anyway 2

  georgemccabe

  committed
  5c833c8

• added docker push for creating new truth data volume

  georgemccabe

  committed
  4f92d45

• set DOCKERHUB_REPO as global env var for workflow instead of making is dynamic

  georgemccabe

  committed
  740cbf0

• Do not fail diff test if there are differences when updating the -ref truth data. See comment for more info

  georgemccabe

  committed
  29d71d7

• Testing: attempt 2 to update truth data

  georgemccabe

  committed
  13dc7dd

• reset default job controls after running test

  georgemccabe

  committed
  00cee88

• rearrange job rules to force behavior for -ref branches, set required pre-req jobs based on what is requested to avoid needing to specify duplicate settings

  georgemccabe

  committed
  a4cff20

• log into docker to push truth data

  georgemccabe

  committed
  c6954e0

• if truth data is not yet available for version (when first run of main_X.Y-ref branch), then use develop data for diff tests

  georgemccabe

  committed
  d2b3010

• set truth data version based on -ref branch to allow diff tests and truth data generate for other versions than develop

  georgemccabe

  committed
  685eead

• Testing: 3rd try - temporarily set to update truth and write to test truth data docker volume

  georgemccabe

  committed
  01cab47

• changed defaults back

  georgemccabe

  committed
  1415b51

• set truth data version to main_vX.Y if running on main_vX.Y push

  georgemccabe

  committed
  d2d8a11

• added input data version variable that can be used to change the input data from develop if needed

  georgemccabe

  committed
  dc5efea

• Per #1546, break unit_ref_config.xml out into 6 different XML's so that we can run more tasks concurrently and complete the unit tests faster via GitHub actions.

  JohnHalleyGotway

  committed
  16184c8

• Merge branch 'feature_1546_ci_testing' into feature_1546_unit_xmls

  JohnHalleyGotway

  committed
  e22d755

• Per #1543, update job control for splitting up the ascii2nc, pb2nc, and ref_config unit tests. ci-run-unit

  JohnHalleyGotway

  committed
  1c75983

• Per #1546, rename unit_test.yml file.

  JohnHalleyGotway

  committed
  0280c3e

• Per #1546 ci-run-unit, group short running unit tests together in the 1a and ref_config jobs to consume fewer machines, hoping that provides resources for the 2a and 2b jobs to start sooner.

  JohnHalleyGotway

  committed
  a931c0b

• Per #1546, re-group the ref_config lead times from 2 to 3. Leads 00, 12, 24, 36, and 48 run 8, 11, 11, 14, and 11 tests, respectively. So do lead 36 by itself as a 3rd job.

  JohnHalleyGotway

  committed
  5624ec7

• Per #1546, split unit_stat_analysis.xml out into 4 separate XML's with tighter dependencies. Elimate Unit Tests group 3a and have each stat_analysis XML run in group 2a once it dependent tests comp…

  Show description for d4bba5d
  JohnHalleyGotway

  committed
  d4bba5d

• Per #1543, ci-run-unit split unit_climatology.xml up into 3 pieces where tests are grouped by the type of climo data being used. If/when we subset the input datasets, this would be a logic grouping…

  Show description for 1b842c2
  JohnHalleyGotway

  committed
  1b842c2

• Per #1546, ci-run-unit fix typo in the climatology_2.0deg name which should be 2.5deg instead.

  JohnHalleyGotway

  committed
  002b22c

• added option to skip obtaining input data volume if it is not needed, ci-run-unit

  georgemccabe

  committed
  fcef721

• fixed typo in xml file name

  georgemccabe

  committed
  c47c35d

• fixed typo, ci-run-unit

  georgemccabe

  committed
  fec3d3d

Commits on Jan 27, 2022

• Merge branch 'develop' into feature_1546_unit_xmls

  georgemccabe

  committed
  f9a6384

• remove workflow because unit_tests.yml performs the compilation test on push events

  georgemccabe

  committed
  2547f99

• Per #1546, added requirement to test XML

  georgemccabe

  committed
  e3cc7e9

• changed script timing info to use datetime instead of time to fix error if using python 3.6+

  georgemccabe

  committed
  5af8ded

• Per #1546, echo value of MET_PYTHON_EXE in tests that use the env var so that an error will occur if MET_TEST_MET_PYTHON_EXE is set when running a test that needs it to be set

  georgemccabe

  committed
  38e7bdd

• Per #1546, fix typo in echo statements.

  JohnHalleyGotway

  authored
  c2688b5

• Per #1546, update unit tests to trigger on pull_request synchronize.

  JohnHalleyGotway

  authored
  cafe079