Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compy integration tests #284

Merged
merged 1 commit into from
Jul 27, 2022
Merged

Compy integration tests #284

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 1 addition & 3 deletions docs/source/dev_guide/testing.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,4 @@ Automated tests

We have a :ref:`GitHub Actions <ci-cd>` Continuous Integration / Continuous Delivery (CI/CD) workflow.

The unit tests are run automatically as part of this. As mentioned earlier,
integration tests must be run on an LCRC machine.

The unit tests are run automatically as part of this. Integration tests must be run on a machine specified above.
123 changes: 0 additions & 123 deletions examples/example_generic.cfg
View file
Open in desktop

This file was deleted.

80 changes: 0 additions & 80 deletions examples/generate_examples.py
View file
Open in desktop

This file was deleted.

91 changes: 91 additions & 0 deletions tests/integration/generated/directions_compy.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
# Testing directions for compy

## Commands to run before running integration tests

### test_bundles

```
rm -rf /compyfs/www//fors729/zppy_test_bundles_www/v2.LR.historical_0201
rm -rf /compyfs/fors729/zppy_test_bundles_output/v2.LR.historical_0201/post
# Generate cfg
python tests/integration/utils.py
zppy -c tests/integration/generated/test_bundles.cfg
# bundle1 and bundle2 should run. After they finish, invoke zppy again to resolve remaining dependencies:
zppy -c tests/integration/generated/test_bundles.cfg
# bundle3 and ilamb should run
```

### test_complete_run

```
rm -rf /compyfs/www//fors729/zppy_test_complete_run_www/v2.LR.historical_0201
rm -rf /compyfs/fors729/zppy_test_complete_run_output/v2.LR.historical_0201/post
# Generate cfg
python tests/integration/utils.py
zppy -c tests/integration/generated/test_complete_run.cfg
```

## Commands to run to replace outdated expected files

### test_bash_generation

```
rm -rf /compyfs/www/zppy_test_resources/expected_bash_files
cd <top level of zppy repo>
# Your output will now become the new expectation.
# You can just move (i.e., not copy) the output since re-running this test will re-generate the output.
mv test_bash_generation_output/post/scripts /compyfs/www/zppy_test_resources/expected_bash_files
# Rerun test
python -u -m unittest tests/integration/test_bash_generation.py
```

#### test_bundles

```
rm -rf /compyfs/www/zppy_test_resources/expected_bundles
# Your output will now become the new expectation.
# Copy output so you don't have to rerun zppy to generate the output.
cp -r /compyfs/www//fors729/zppy_test_bundles_www/v2.LR.historical_0201 /compyfs/www/zppy_test_resources/expected_bundles
mkdir -p /compyfs/www/zppy_test_resources/expected_bundles/bundle_files
cp -r /compyfs/fors729/zppy_test_bundles_output/v2.LR.historical_0201/post/scripts/bundle*.bash /compyfs/www/zppy_test_resources/expected_bundles/bundle_files
cd /compyfs/www/zppy_test_resources/expected_bundles
# This file will list all the expected images.
find . -type f -name '*.png' > ../image_list_expected_bundles.txt
cd <top level of zppy repo>
# Rerun test
python -u -m unittest tests/integration/test_bundles.py
```

### test_campaign

```
cd <top level of zppy repo>
./tests/integration/generated/update_campaign_expected_files_compy.sh
```

### test_complete_run

```
rm -rf /compyfs/www/zppy_test_resources/expected_complete_run
# Your output will now become the new expectation.
# Copy output so you don't have to rerun zppy to generate the output.
cp -r /compyfs/www//fors729/zppy_test_complete_run_www/v2.LR.historical_0201 /compyfs/www/zppy_test_resources/expected_complete_run
cd /compyfs/www/zppy_test_resources/expected_complete_run
# This file will list all the expected images.
find . -type f -name '*.png' > ../image_list_expected_complete_run.txt
cd <top level of zppy repo>
# Rerun test
python -u -m unittest tests/integration/test_complete_run.py
```

### test_defaults

```
rm -rf /compyfs/www/zppy_test_resources/test_defaults_expected_files
mkdir -p /compyfs/www/zppy_test_resources/test_defaults_expected_files
# Your output will now become the new expectation.
# You can just move (i.e., not copy) the output since re-running this test will re-generate the output.
mv test_defaults_output/post/scripts/*.settings /compyfs/www/zppy_test_resources/test_defaults_expected_files
# Rerun test
python -u -m unittest tests/integration/test_defaults.py
```
16 changes: 16 additions & 0 deletions tests/integration/generated/update_campaign_expected_files_compy.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Run this script to update expected files for test_campaign.py
# Run from the top level of the zppy repo
# Run as `./tests/integration/update_campaign_expected_files.sh`

for campaign in "cryosphere" "cryosphere_override" "high_res_v1" "none" "water_cycle" "water_cycle_override"
do
echo ${campaign}
rm -rf /compyfs/www/zppy_test_resources/test_campaign_${campaign}_expected_files
mkdir -p /compyfs/www/zppy_test_resources/test_campaign_${campaign}_expected_files
# Your output will now become the new expectation.
# You can just move (i.e., not copy) the output since re-running this test will re-generate the output.
mv test_campaign_${campaign}_output/post/scripts/*.settings /compyfs/www/zppy_test_resources/test_campaign_${campaign}_expected_files
done

# Rerun test
python -m unittest tests/integration/test_campaign.py
3 changes: 2 additions & 1 deletion tests/integration/template_bundles.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@ input_subdir = archive/atm/hist
mapping_file = "#expand mapping_path#map_ne30pg2_to_cmip6_180x360_aave.20200201.nc"
# To run this test, edit `output` and `www` in this file, along with `actual_images_dir` in test_bundles.py
output = "#expand user_output#zppy_test_bundles_output/v2.LR.historical_0201"
partition = debug
partition = "#expand partition_short#"
qos = "#expand qos_short#"
walltime = "02:00:00"
www = "#expand user_www#zppy_test_bundles_www"

Expand Down
14 changes: 10 additions & 4 deletions tests/integration/template_complete_run.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@ input_subdir = archive/atm/hist
mapping_file = "#expand mapping_path#map_ne30pg2_to_cmip6_180x360_aave.20200201.nc"
# To run this test, edit `output` and `www` in this file, along with `actual_images_dir` in test_complete_run.py
output = "#expand user_output#zppy_test_complete_run_output/v2.LR.historical_0201"
partition = debug
partition = "#expand partition_short#"
qos = "#expand qos_short#"
www = "#expand user_www#zppy_test_complete_run_www"

[climo]
Expand Down Expand Up @@ -86,9 +87,10 @@ years = "1850:1854:2", "1850:1854:4",
[[ atm_monthly_180x360_aave ]]
climo_diurnal_frequency = "diurnal_8xdaily"
climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave"
partition = compute
partition = "#expand partition_long#"
qos = "#expand qos_long#"
sets = "lat_lon","zonal_mean_xy","zonal_mean_2d","polar","cosp_histogram","meridional_mean_2d","enso_diags","qbo","diurnal_cycle","annual_cycle_zonal_mean","streamflow", "zonal_mean_2d_stratosphere",
walltime = "01:00:00"
walltime = "#expand diags_walltime#"

[[ atm_monthly_180x360_aave_environment_commands ]]
environment_commands = "#expand diags_environment_commands#"
Expand All @@ -108,6 +110,8 @@ years = "1850:1854:2", "1850:1854:4",
climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave"
climo_subsection = "atm_monthly_180x360_aave"
diff_title = "Difference"
partition = "#expand partition_long#"
qos = "#expand qos_long#"
ref_final_yr = 1851
ref_name = "v2.LR.historical_0201"
ref_start_yr = 1850
Expand All @@ -119,6 +123,7 @@ years = "1850:1854:2", "1850:1854:4",
tag = "model_vs_model"
ts_num_years_ref = 2
ts_subsection = "atm_monthly_180x360_aave"
walltime = "#expand diags_walltime#"
years = "1852-1853",

[mpas_analysis]
Expand All @@ -128,7 +133,8 @@ climo_years ="1850-1854", "1855-1860",
enso_years = "1850-1854", "1855-1860",
mesh = "EC30to60E2r2"
parallelTaskCount = 6
partition = compute
partition = "#expand partition_long#"
qos = "#expand qos_long#"
ts_years = "1850-1854", "1850-1860",
walltime = "00:30:00"

Expand Down
2 changes: 1 addition & 1 deletion tests/integration/template_directions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ rm -rf #expand expected_dir#expected_bundles
cp -r #expand user_www#zppy_test_bundles_www/v2.LR.historical_0201 #expand expected_dir#expected_bundles
mkdir -p #expand expected_dir#expected_bundles/bundle_files
cp -r #expand user_output#zppy_test_bundles_output/v2.LR.historical_0201/post/scripts/bundle*.bash #expand expected_dir#expected_bundles/bundle_files
cd #expected_bundles
cd #expand expected_dir#expected_bundles
# This file will list all the expected images.
find . -type f -name '*.png' > ../image_list_expected_bundles.txt
cd <top level of zppy repo>
Expand Down
Loading