Remove redundant response of empty result in AsyncShardFetch to avoid OOM issue

[maosuhan]

In the process of full restart, OOM problems are often encountered. After dump analysis and we found the following problems.

In the process of fetching data, the master will ask each data whether it owns the shard, and all the returned results of the node will be saved in the map<nodeId, nodeResponse> in AsyncShardFetch. In most cases, a shard only has data on several nodes, but the intermediate result map will still save the return results of all nodes.

In this MR, we only save valid intermediate result of node responses and ignore the node responses that does not hold the shard at all.

It is proved to be successfully in our company and the OOM issue is gone after the optimization.

[cla-checker-service]

❌ Author of the following commits did not sign a Contributor Agreement:
e37d3e1

Please, read and sign the above mentioned agreement if you want to contribute to this project

[DaveCTurner]

What version are you using @maosuhan? I think this excessive memory usage was addressed by #77991.

[maosuhan]

@DaveCTurner We already backport the #77991, but the OOM issue still exists.
We use 7.6.2 in production. The shard number is 1 million, and the nodes number is nearly 400.

The main part which occupy the memory is not DiscoveryNode but StoreFilesMetadata which is a node response content and is relatively large.

[DaveCTurner]

Can you help me understand why an empty StoreFilesMetadata is still "relatively large". I get that there are a lot of them, but each one should be small. Is that not the case?

I think this will be addressed by some bigger changes related to #77466, particularly #80694 and friends.

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[DaveCTurner]

Also would you sign the CLA?

[maosuhan]

CLA Done @DaveCTurner

[maosuhan]

@DaveCTurner As you can see in the screenshot, there was totally 32927024 objets in the dump and they occupied more than 15GB.
So each object size is 0.45KB.

[DaveCTurner]

So each object size is 0.45KB.

i.e. 450 bytes. That's a lot more than I expect. Can you explain why it's so much?

[DaveCTurner]

Are you sure you signed the CLA with the address [email protected] attached to your commits? I don't see this address in the list.

[DaveCTurner]

On reflection I'd rather avoid allocating these objects entirely rather than creating them and then filtering them out as proposed here. #84034 should do what we want.

[maosuhan]

Thanks for your effort! I think it is more reasonable to do that.
I just debugged in IDE, below is the screenshot.

metadata and commitUserData are all empty HashMap which is read through streamInput. And each hashmap will occupy 64 bytes each. So the object is at least 128 bytes.
The non-empty result is definitely much larger than the empty ones but the accumulated wasted memory will be very large if the cluster and node number are both large.
Feel free to close this issue.

[DaveCTurner]

Sounds good, thanks for raising the issue. Closing this, will proceed with #84034 instead.

[elasticsearchmachine]

@maosuhan please enable the option "Allow edits and access to secrets by maintainers" on your PR. For more information, see the documentation.