Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix attribute array comparison #6181

Merged
merged 11 commits into from
Oct 25, 2024

Conversation

ukmo-ccbunney
Copy link
Contributor

@ukmo-ccbunney ukmo-ccbunney commented Oct 18, 2024

🚀 Pull Request

Description

Fixes: #6027

Logic for comparing arrays in attributes was relying on old numpy behaviour where an array compared to another any object would return a scalar True or False. Current numpy behaviour is to do a element-wise comparison which throws an error if the arrays are of different size.

Fix is to use the numpy.array_equal.


Consult Iris pull request check list

@ukmo-ccbunney ukmo-ccbunney self-assigned this Oct 18, 2024
Copy link

codecov bot commented Oct 18, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.83%. Comparing base (8c213f0) to head (50b624d).
Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #6181      +/-   ##
==========================================
- Coverage   89.83%   89.83%   -0.01%     
==========================================
  Files          88       88              
  Lines       23211    23208       -3     
  Branches     4316     4316              
==========================================
- Hits        20852    20849       -3     
  Misses       1627     1627              
  Partials      732      732              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@ukmo-ccbunney
Copy link
Contributor Author

ukmo-ccbunney commented Oct 18, 2024

Note, this change fixes what I believe is a bug here:

iris/lib/iris/_merge.py

Lines 388 to 391 in d3071ff

diff_attrs = [
repr(key[1])
for key in attrs_1
if np.all(attrs_1[key] != attrs_2[key])

Line 391 should be checking for any differences in the array elements, not all.

It also cleans up a now unnecessary try...except block that I believe was only needed for older versions of numpy.

try:
match = bool(match)
except ValueError:
match = match.all()

I have extended the existing unit test to check for correct handling inequality.
However, I question the validity of this test

values = dict(one=np.arange(1), two=np.arange(1), three=np.arange(1))
left = LimitedAttributeDict(dict(one=0, two=0, three=0))
right = LimitedAttributeDict(**values)
self.assertEqual(left, right)
self.assertEqual(left, values)

which now fails as it is testing a scalar value against a single element array. With the old logic, this would work, but it is flawed as np.all( np.array([0]) == 0 ) and np.all( np.zeros(100) == 0 ) would both return True, even though the latter is not what is intended.

I have commented out this test at the moment and would appreciate the opinion of @pp-mo.

…bunney/iris into bugfix/cube_attr_compare

* 'bugfix/cube_attr_compare' of https://github.com/ukmo-ccbunney/iris:
  [pre-commit.ci] auto fixes from pre-commit.com hooks
pp-mo
pp-mo previously requested changes Oct 21, 2024
Copy link
Member

@pp-mo pp-mo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comment where I think this changes too much from as-is behaviour.
Otherwise all good , I think

…mpare

* upstream/main:
  Document use of new_axis to control merge (SciTools#6180)
  Updated environment lockfiles (SciTools#6184)
  [pre-commit.ci] pre-commit autoupdate (SciTools#6175)
  Updated environment lockfiles (SciTools#6183)
@ukmo-ccbunney ukmo-ccbunney marked this pull request as ready for review October 23, 2024 15:08
@ESadek-MO ESadek-MO self-requested a review October 25, 2024 12:08
Copy link
Contributor

@ESadek-MO ESadek-MO left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to build on top of @pp-mo 's review and consider this good to go!
Thanks @ukmo-ccbunney !

@ESadek-MO ESadek-MO dismissed pp-mo’s stale review October 25, 2024 12:17

@pp-mo is currently unavailable to approve change

@ESadek-MO ESadek-MO merged commit f58f050 into SciTools:main Oct 25, 2024
21 checks passed
@ukmo-ccbunney ukmo-ccbunney deleted the bugfix/cube_attr_compare branch October 25, 2024 16:05
HGWright added a commit to HGWright/iris that referenced this pull request Oct 31, 2024
* upstream/main: (194 commits)
  Restore latest Whats New files.
  [pre-commit.ci] pre-commit autoupdate (SciTools#6205)
  correct major minor in whatsnew (SciTools#6202)
  What's new updates for v3.11.0rc0 . (SciTools#6201)
  Update CF standard names table. (SciTools#6200)
  Specify meta in dask.array.map_blocks (SciTools#5989)
  added in a vertical rule for surface fields (SciTools#5734)
  Updated environment lockfiles (SciTools#6197)
  Reduce duplication in cf.py spanning checks. (SciTools#6196)
  Fix attribute array comparison (SciTools#6181)
  Enable factory references to create new dimensions on load. (SciTools#6168) (SciTools#6194)
  Improve handling of masks in concatenate (SciTools#6187)
  Demo Numpy v2 (SciTools#6035)
  Bump scitools/workflows from 2024.10.1 to 2024.10.2 (SciTools#6186)
  Document use of new_axis to control merge (SciTools#6180)
  Updated environment lockfiles (SciTools#6184)
  [pre-commit.ci] pre-commit autoupdate (SciTools#6175)
  Updated environment lockfiles (SciTools#6183)
  Update to `cell_method` parsing (SciTools#6083)
  Bump scitools/workflows from 2024.10.0 to 2024.10.1 (SciTools#6179)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

Cannot load cubes that contain multiple variables with array-type attributes with different sizes
3 participants