[#2282] improvement(spark3): Invoke with object self in DelegationRssShuffleManager

[zuston]

What changes were proposed in this pull request?

Fix the bug of none object self when invoking the reflection method.

Why are the changes needed?

Fix: #2282

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existing tests

[jerqi]

What will it happen if we don't have this?

[github-actions]

Test Results

 2 966 files  +23   2 966 suites  +23   6h 32m 32s ⏱️ + 13m 34s
 1 096 tests ± 0   1 094 ✅ + 1   2 💤 ±0  0 ❌  - 1 
13 735 runs  +53  13 705 ✅ +54  30 💤 ±0  0 ❌  - 1 

Results for commit 4afa783. ± Comparison against base commit 0552d3b.

[zuston]

What will it happen if we don't have this?

Nothing will happen, because we bypass the getReader in the DelegationRssShuffleManager, so the acutall getReader invoking will happen in the underlying concrate shuffle manager. But this is an obvious bug, if someone wants to implement custom shuffle manager, he will follow the incorrect examples.

[jerqi]

What will it happen if we don't have this?

Nothing will happen, because we bypass the getReader in the DelegationRssShuffleManager, so the acutall getReader invoking will happen in the underlying concrate shuffle manager. But this is an obvious bug, if someone wants to implement custom shuffle manager, he will follow the incorrect examples.

So I think this is an improvement instead of a bug fix. If this is bug fix, we would better have a ut for it.

[maobaolong]

@zuston Good catch! We encountered this issue too in this week! Thanks for your contribution!

[wForget]

Nothing will happen, because we bypass the getReader in the DelegationRssShuffleManager, so the acutall getReader invoking will happen in the underlying concrate shuffle manager. But this is an obvious bug, if someone wants to implement custom shuffle manager, he will follow the incorrect examples.

I prefer to infer the appropriate shuffle manager from ShuffleHandle rather than overriding the shuffle manager configuration on the dirver side, so that we can implement stage-level fallback.