Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

👌🚇 Improve result comparison threshold for spectra #937

Closed
jsnel opened this issue Dec 12, 2021 · 0 comments · Fixed by #936 or #938
Closed

👌🚇 Improve result comparison threshold for spectra #937

jsnel opened this issue Dec 12, 2021 · 0 comments · Fixed by #936 or #938

Comments

@jsnel
Copy link
Member

jsnel commented Dec 12, 2021

Issue identified in #936, comment reproduced below.

After the update we find that the result comparison still fails but only for a singular data point in one of the examples (log below). Digging a little deeper (looking at the plot) we find that the point of failure is at a place were the spectral value in question is very small (~-1.497), compared to the overall maximum of the spectrum (~7500). However, the spectra are compared locally (per wavelength) and the absolute tolerance for the comparison between results is scaled with the local value of the data, meaning that the tolerance for a point close to zero is much tighter than the tolerance for a point near the maximum. This is a point where the result comparison could be further refined, but for now we accept this as a PASS ✔️ and this PR may be merged.

FAILED .github/test_result_consistency.py::test_result_data_var_consistency[study_fluorescence-decay_associated_spectra]
E       AssertionError: Result data_var data mismatch: 'decay_associated_spectra' in 'dataset1.nc'.
E         With sum of absolute difference: 0.0039020775839004873 and shape: (49, 5)
E         Mean difference: 1.592684728122648e-05
E         
E       assert False
E        +  where False = <function allclose.<locals>._allclose at 0x7f40599e5790>(<xarray.DataArray 'decay_associated_spectra' (spectral: 49, component: 5)>\narray([[-145.581345,   52.590453,   21.6330...7\n    rate      (component) float64 ...\n    lifetime  (component) float64 ...\nDimensions without coordinates: component, <xarray.DataArray 'decay_associated_spectra' (spectral: 49, component: 5)>\narray([[-145.581346,   52.590453,   21.6330...7\n    rate      (component) float64 ...\n    lifetime  (component) float64 ...\nDimensions without coordinates: component, atol=array([[1.73546487e-05, 6.26927054e-06, 2.57886253e-06, 2.17881660e-06,\n        6.32769907e-06],\n       [1.50005268e-0...     5.91504505e-06],\n       [5.92400043e-05, 4.45607905e-05, 4.22466039e-06, 1.01596965e-04,\n        6.42848991e-06]]), rtol=1e-05, print_fail=20)

.github/test_result_consistency.py:221: AssertionError
----------------------------- Captured stdout call -----------------------------
allclose first 1 failures:
  (40, 1): -1.4969902223364386 -1.4969635695838823
=========================== short test summary info ============================
FAILED .github/test_result_consistency.py::test_result_data_var_consistency[study_fluorescence-decay_associated_spectra]

Originally posted by @jsnel in #935 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant