Split large batches of documents if received 413 from Elasticsearch

[rdner]

Describe the enhancement:

Currently, after seeing a 413 response from Elasticsearch the whole batch is dropped and the error is logged (#29368). Some of our customers would like to preserve at least some data from the batch instead of discarding the whole batch.

The proposal is:

• When seeing a 413 response from Elasticsearch try to split the current batch (maybe in 2, or based on size of each document in the batch)
• If the 413 response is seen again – repeat the process until:
  • either all the smaller (split) batches are successfully sent to Elasticsearch or
  • the initial batch is reduced to a single document that cannot pass the http.max_content_length threshold in Elasticsearch
• If a batch contain only a single document that cannot be uploaded – drop the batch

Something similar was done in this PR logstash-plugins/logstash-output-elasticsearch#497

Please ensure that each of these actions are logged, in particular:

when the batch is dropped, please state in the log for info:

• inform the smaller batch was dropped
• how many iterations it took to reduce the size (if this is possible)
• Any info from the batch that was dropped (ideally if we know what application or integration)

When the batch is being cut to size:

• what is current size, and what is it being split into
• what is the current configured max_bulk_size

Describe a specific use case for the enhancement or feature:

Some of our clients are more sensitive to data loss than others and this enhancement would allow to preserve more data in case of misconfiguration of http.max_content_length in Elasticsearch or bulk_max_size in beats. This would improve the situation in most of the cases but it would not completely solve the data loss problem.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[jlind23]

ping @nimarezainia for prioritization and awareness

[nimarezainia]

@rdner this sounds like the right thing to do and makes our product more robust. What would be the level of effort involved in getting this done?

Ideally the buffers would match dynamically so we wouldn't hit these issues but I know that is near impossible.

Could you please ensure that each of these actions are logged, in particular:

when the batch is dropped, please state in the log for info:

• inform the smaller batch was dropped
• how many iterations it took to reduce the size (if this is possible)
• Any info from the batch that was dropped (ideally if we know what application or integration)

When the batch is being cut to size:

• what is current size, and what is it being split into
• what is the current configured max_bulk_size

[rdner]

@nimarezainia
We sure need to log every single step, I totally agree with that, I copied it into the description.

Regarding the estimation of effort, I'm quite new to the project, so it's hard for me to give a precise estimation on the effort, I would ask @faec for help here since we already touched on this topic once.

@cmacknz it might be worth considering to introduce this kind of behaviour into the shippers design too.

[jlind23]

@cmacknz Should I "close" this one and focus on the shippers as you have already included it in the V2 implementation?

[cmacknz]

Let's keep the issue as it is a good description of the work to do. We could remove the release target and labels though. This will happen as part of the shipper work at some to be determined point in the future.

[Foxboron]

Is this not going to be fixed in the current beat implementation?

[jlind23]

@Foxboron Even if here we talk about fixing it in the shippers it doesn't mean that it will not be fixed in standalone beats.
The work that will be done under the shipper project but doesn't exclude beats fix.

[cmacknz]

The current plan is to address this in Beats so the fix is available sooner, and then port it into shipper afterwards so we aren't tied to date when the shipper is ready to be released.

[amitkanfer]

Ideally the buffers would match dynamically so we wouldn't hit these issues but I know that is near impossible.

@nimarezainia why is it near impossible? is it because Agent / beat can send to multiple ES clusters with different size limits? I agree with the conclusion, just want to make sure i understand all the reasons for it.

[nimarezainia]

Ideally the buffers would match dynamically so we wouldn't hit these issues but I know that is near impossible.

@nimarezainia why is it near impossible? is it because Agent / beat can send to multiple ES clusters with different size limits? I agree with the conclusion, just want to make sure i understand all the reasons for it.

I believe I was told that the ES buffer size is not known to us. this may have changed. if there was an API for us to read that, perhaps our output can be set to match, minimizing drops. perhaps things have changed since that comment.