Limit shard failures accumulated by searches

[DaveCTurner]

Today we create an array of size getNumShards() to collect shard failures received during a search (on the first failure):

elasticsearch/server/src/main/java/org/elasticsearch/action/search/AbstractSearchAsyncAction.java

Line 541 in f3a4813

shardFailures = new AtomicArray<>(getNumShards());

At high shard counts this array itself could be pretty large, and if a substantial number of shards fail then its contents will become massive. I'm pretty sure we don't need to accumulate all the failures in this situation, they're almost certainly not useful for the end-user or even for debugging. Could we instead just just track the first few failures and keep a count of the rest?

Relates #77466

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)