Use the correct chunktype for Dask arrays

[bouweandela]

🚀 Pull Request

Description

Use the correct chunktype for Dask arrays. Closes #5800.

---

Consult Iris pull request check list

---

Add any of the below labels to trigger actions on this PR:

• benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts

[pp-mo]

Hi @bouweandela
The existing test breakage is a fairly minor matter, but I think it indicates that removing line 287b is causing the da.from_array to do trial access on the data -- so I think it is probably better to continue to provide a meta key in this case.

However, I am also concerned that ideally we should also be addressing #5800 if possible (!)
In which case, I think this should have a meta of type np.ma.MaskedArray in place of np.ndarray, when appropriate.

From my reading of dask docs, I think it may also be better to pass a zero-shaped instance instead of a class, since the from_array docs don't actually mandate that usage.
Also, it actually doesn't work if I pass meta=np.ma.MaskedArray : This gives a TypeError: MaskedArray.__new__() got an unexpected keyword argument 'shape'.

So I think something like da.from_array(real, meta=real.__class__((), dtype=real.dtype)) works here, and is probably a better way.

Does this make sense to you ?

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.78%. Comparing base (7c313ff) to head (00af67e).
Report is 21 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5801      +/-   ##
==========================================
+ Coverage   89.76%   89.78%   +0.01%     
==========================================
  Files          93       93              
  Lines       22982    23004      +22     
  Branches     5006     5015       +9     
==========================================
+ Hits        20630    20654      +24     
+ Misses       1622     1620       -2     
  Partials      730      730              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[bouweandela]

@bouweandela I suspect that the pre-commit failure is unrelated to this pull request. When I run pre-commit run --all-files on the main branch I get the same error.

Yes, puzzling. We are looking into it...

[pp-mo]

@bouweandela I suspect that the pre-commit failure is unrelated to this pull request. When I run pre-commit run --all-files on the main branch I get the same error.

Yes, puzzling. We are looking into it...

Ok I think we know what this is now -- fix coming soon.
It's only "odd" because the flake8 error code is bugged !

UPDATE: see #5925 -- we think this fixes it

[pp-mo]

I think everything here is basically good, and it's a really nice touch to provide a fix for concatenate as well as stack.

However, I think we need to rethink the merge, since some changes from #5588 seem to have got lost.
I guess that means that merging back into this will cause problems
( I expect at least some tests ought then to fail, otherwise where is the testing of those new keywords ?? )
So I haven't updated with main, as you may want to resolve this in your own way.

From a quick search I also think that, unlike 'stack', there are multiple uses of 'concatenate' elsewhere in the Iris code that would benefit from using the new routine.
However, given how involved this is getting, it would make good sense to get this working + merged in first : then I can follow up with another to expand the use of _lazy.data.concatenate.

[bouweandela]

Thank you for reviewing @pp-mo! I just merged the latest main into this branch and the pre-commit test is passing fine now 🥳

However, I think we need to rethink the merge, since some changes from #5588 seem to have got lost.
I guess that means that merging back into this will cause problems
( I expect at least some tests ought then to fail, otherwise where is the testing of those new keywords ?? )
So I haven't updated with main, as you may want to resolve this in your own way.

I refactored the as_lazy_data function a bit, but the functionality is still there (see #5801 (comment)). If you prefer, I can undo that?

[pp-mo]

If you prefer, I can undo that?

Thanks no, I just hadn't understood fully what you were doing. It's good !