Release 2022.9.2

[jrbourbeau]

Release version	2022.9.2
Planned release date	2022-09-30
Status	On-track
Release manager	@jrbourbeau

Best effort

Try to close before the release but will not block the release

• Catch BaseException on UCX read error distributed#6996
• test_serialize_numba: Workaround issue with np.empty_like in NumPy 1.23 distributed#7089
• Refactor occupancy distributed#7075

Blocker

Issues that would cause us to block and postpone the release if not fixed

• ...
• ...

Comments

cc @ian-r-rose @gjoseph92 @crusaderky @quasiben @jakirkham

[quasiben]

We'd like to include dask/distributed#6996 as part of the release. It's limited to changes within UCX usage in Dask

[quasiben]

Graham also fixed the CI issue we saw with numba serialization:
dask/distributed#7089

[hendrikmakait]

I'd love to see dask/distributed#7075 in that release.

[jrbourbeau]

We'd like to include dask/distributed#6996 as part of the release. It's limited to changes within UCX usage in Dask

Sounds good. @quasiben could I ask you to shepherd those changes through?

I'd love to see dask/distributed#7075 in that release.

I just pinged Gabe and Guido to see if they have bandwidth to review

[gjoseph92]

@hendrikmakait can you remind me, have we run benchmarking on dask/distributed#7075 and/or are we confident there are no behavioral changes, just a different way of deriving an identical occupancy number? I've started reviewing and though everything looks fine, it's just a substantial change to a delicate part of scheduling (occupancy). I'd be slightly wary of releasing it without some integration testing first.

[quasiben]

Sounds good. @quasiben could I ask you to shepherd those changes through?

+1

[crusaderky]

@hendrikmakait can you remind me, have we run benchmarking on dask/distributed#7075 and/or are we confident there are no behavioral changes, just a different way of deriving an identical occupancy number? I've started reviewing and though everything looks fine, it's just a substantial change to a delicate part of scheduling (occupancy). I'd be slightly wary of releasing it without some integration testing first.

Agreed, some A/B testing would make me feel a lot more comfortable.

[hendrikmakait]

@hendrikmakait can you remind me, have we run benchmarking on dask/distributed#7075 and/or are we confident there are no behavioral changes, just a different way of deriving an identical occupancy number? I've started reviewing and though everything looks fine, it's just a substantial change to a delicate part of scheduling (occupancy). I'd be slightly wary of releasing it without some integration testing first.

IIRC, @fjetter did some preliminary performance evaluation on #7030, but I am not sure how much has changed since. I agree, running benchmarks after sorting out code review issues should happen before merging.

With respect to behavioral changes, I think we are pretty much guaranteed to have some (e.g., we do not have the non-deterministic recalculation of occupancy anymore). The question should rather be whether these impact performance.

[hendrikmakait]

Update: We are currently running an A/B test to assess dask/distributed#7075, but we should not hold up the release waiting for it.

[jrbourbeau]

Okay, I'm going to start pushing out the release. The two PRs @quasiben mentioned as in -- let's include dask/distributed#7075 in the next release (apologies @hendrikmakait)

[quasiben]

+1 Thanks @jrbourbeau !

[jrbourbeau]

2022.9.2 is out on PyPI

[jrbourbeau]

conda-forge package and docker images are published. Thanks all!