🧪🚇 Add integration test result validation#754

Merged

jsnel merged 29 commits intoglotaran:stagingfrom s-weigand:compare-results

Sep 5, 2021

+487-2

Commits on Sep 4, 2021

🧪 Added result consistency test script

s-weigand
authored and
jsnel
committed
🚇 Added result consistency testing to the integration test workflow

s-weigand
authored and
jsnel
committed
👌 Propagate git interaction errors to inform users

s-weigand
authored and
jsnel
committed
🔧 Add pytest-allclose for more usefull error reporting

s-weigand
authored and
jsnel
committed
👌 Use colored pytest output for better readability

s-weigand
authored and
jsnel
committed
👌 Show the commits used to create the results

s-weigand
authored and
jsnel
committed
👌 Added tests for optimized parameters

s-weigand
authored and
jsnel
committed
🧹 Renamed variables used for comparing to expected and current

s-weigand
authored and
jsnel
committed
👌 Print up to 20 different values if "allclose" fails

s-weigand
authored and
jsnel
committed
Adjust tolerances to pass on CI if reference and current are the same
jsnel
committed
👌 Make git interactions more save by passing the folder to use to git

s-weigand
authored and
jsnel
committed
👌 Implemented 'EXAMPLE_BLOCKLIST' and added 'transient_absorption_two_dataset'

s-weigand
authored and
jsnel
committed
👌 Only lower absolute tolerance for SVD comparison

s-weigand
authored and
jsnel
committed
👌 Added 'ex_spectral_guidance' to EXAMPLE_BLOCKLIST

s-weigand
authored and
jsnel
committed
📚 Added instructions to locally run result consistency test

s-weigand
authored and
jsnel
committed
👌 Added dataset file name to report on failing test

s-weigand
authored and
jsnel
committed
👌 Added data_var name to error reporting

s-weigand
authored and
jsnel
committed
👌 Special cased missing weighted_data

s-weigand
authored and
jsnel
committed
👌 Added label displaying on test_result_parameter_consistency fail

s-weigand
authored and
jsnel
committed
🩹 Fixed line length issue

s-weigand
authored and
jsnel
committed
👌 Use value difference to check data_vars

s-weigand
authored and
jsnel
committed
👌 Swap abs_diff and float_resolution, that way rtol has some effect

s-weigand
authored and
jsnel
committed
👌 Use float32 precission as absolute tolerance

s-weigand
authored and
jsnel
committed
👌 Improved error reporting on failure by showing mean difference

s-weigand
authored and
jsnel
committed
♻️ Refactored data_var tests not to run in a loop but using fixtures

s-weigand
authored and
jsnel
committed
👌 Made epsilon for residual scale with original data

s-weigand
authored and
jsnel
committed
👌 Add option to specify path for local comparison

s-weigand
authored and
jsnel
committed
👌 Allow missing coords in some variables
jsnel
committed

Commits on Sep 5, 2021

🩹 Reorder dimensions before comparison
jsnel
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🧪🚇 Add integration test result validation#754

Commits on Sep 4, 2021

🧪 Added result consistency test script

🚇 Added result consistency testing to the integration test workflow

👌 Propagate git interaction errors to inform users

🔧 Add `pytest-allclose` for more usefull error reporting

👌 Use colored pytest output for better readability

👌 Show the commits used to create the results

👌 Added tests for optimized parameters

🧹 Renamed variables used for comparing to expected and current

👌 Print up to 20 different values if "allclose" fails

Adjust tolerances to pass on CI if reference and current are the same

👌 Make git interactions more save by passing the folder to use to git

👌 Implemented 'EXAMPLE_BLOCKLIST' and added 'transient_absorption_two_dataset'

👌 Only lower absolute tolerance for SVD comparison

👌 Added 'ex_spectral_guidance' to EXAMPLE_BLOCKLIST

📚 Added instructions to locally run result consistency test

👌 Added dataset file name to report on failing test

👌 Added data_var name to error reporting

👌 Special cased missing weighted_data

👌 Added label displaying on test_result_parameter_consistency fail

🩹 Fixed line length issue

👌 Use value difference to check data_vars

👌 Swap abs_diff and float_resolution, that way rtol has some effect

👌 Use float32 precission as absolute tolerance

👌 Improved error reporting on failure by showing mean difference

♻️ Refactored data_var tests not to run in a loop but using fixtures

👌 Made epsilon for residual scale with original data

👌 Add option to specify path for local comparison

👌 Allow missing coords in some variables

Commits on Sep 5, 2021

🩹 Reorder dimensions before comparison