Disagg: Wait tasks in thread pool finished before return

[JinheLin]

What problem does this PR solve?

Issue Number: ref #9378

Problem Summary:

If exception thrown in building read tasks, exception will propagate by the future object, and the request will abort immediately.

tiflash/dbms/src/Storages/StorageDisaggregatedRemote.cpp

Lines 102 to 106 in 9afd0b2

	void waitForCompletion(std::vector<std::future<void>> & futures)
	{
	for (auto & f : futures)
	f.get();
	}

However, there are many tasks running in the thread pool and these task has caught many local variables by reference. If the request aborted before these tasks finished, these reference will be dangling.

What is changed and how it works?

Wait the tasks in the thread pool finished before return like before

tiflash/dbms/src/Storages/StorageDisaggregatedRemote.cpp

Lines 223 to 225 in e6fc04a

	// Let's wait for all threads to finish. Otherwise local variable references will be invalid.
	// The first exception will be thrown out if any, after all threads are finished, which is safe.
	thread_manager->wait();

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

None

[CalvinNeo]

lgtm

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: CalvinNeo, Lloyd-Pottiger

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [CalvinNeo,Lloyd-Pottiger]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2024-08-28 09:27:54.29313986 +0000 UTC m=+948869.427589981: ☑️ agreed by CalvinNeo.
• 2024-08-28 09:59:14.880311224 +0000 UTC m=+950750.014761331: ☑️ agreed by Lloyd-Pottiger.