Cancel background Glue partition futures on failure

[pettyjamesm]

Description

This change ensures that GlueHiveMetastore cancels any ongoing parallel partition loading operations if a failure is encountered in one background thread or if the query is canceled. Previously, these threads would continue to run in the background unnecessarily.

Release notes

(x) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
( ) Release notes are required, with the following suggested text:

[pettyjamesm]

@findepi - this change is ready for review, the test failure was caused by a race-prone assertion in TestThrottledAsyncQueue that depends on wall clock time through the RateLimiter and is unrelated to this change.

[arhimondr]

The change looks good to me. However I wonder why ExecutorCompletionService is necessary? I'm thinking if it would be possible to simply use MoreFutures#allAsListWithCancellationOnFailure from Airlift?

[pettyjamesm]

The change looks good to me. However I wonder why ExecutorCompletionService is necessary? I'm thinking if it would be possible to simply use MoreFutures#allAsListWithCancellationOnFailure from Airlift?

I think the intention is to allow processing of each future as soon as it's available without waiting for all to complete, and since the sub-futures are not pre-sorted in any meaningful way, we don't need the ordering semantics.

[arhimondr]

From offline discussion:

@pettyjamesm Pointed out that futures have to be cancelled not only when one of them fails, but also in an event of a planner thread being interrupted