Failure Store: Update go-docappender to respect failure store status

[1pkg]

Description

This PR updates go-docappender to respect new failure store response status and emit new correspodning metrics accordingly. The old "indexed" metrics stay intact instead a new separate set of failure store labels is exposed.

This PR depends on changes made for BulkIndexerResponseItem in elastic/go-elasticsearch#948 and should only be merged afterwards.

How to test:

Use an instance of ES that has failure store feature enabled and enable failure store for a data stream via component template with.

POST _component_template/$name
{
  "template": {
    "data_stream_options": {
      "failure_store": {
        "enabled": true
      }
    }
  }
}

Set a custom "fail" ingest pipeline with.

PUT _ingest/pipeline/$name
{
  "processors": [
    {
      "fail": {
        "message": "fail"
      }
    }
  ]
}

Ingest some data to corresponding data stream, then check that failure store metrics are getting reported correctly.

To emulate "failed" status, set backing data stream index to be read only with.

PUT /$index/_settings
{
  "index.blocks.read_only": true
}

[axw]

Looks good! Just one question

[axw]

One more question, sorry: should we be measuring "not_enabled" too?

[1pkg]

One more question, sorry: should we be measuring "not_enabled" too?

Agree, this metric could be useful too. Updated to expose it in the last commit.

[marclop]

I think this looks good overall, just one small question

[marclop]

Is it worth adding the FailureStore struct in the legacy stats metrics?

[1pkg]

Correct me if I'm wrong, but per my understanding legacy stats metrics are residing in https://github.com/elastic/go-docappender/blob/main/appender.go#L65. Which I already reverted in 48f9d85.

While this is just generic bulk response container which is used to pass data to appender for reporting OTEL metrics, etc.

[simitt]

I'm not sure how the status: "FailureStore" is supposed to be used?
E.g. when calculating SLOs and being interested in success vs failed documents. If I understand it correct then the failure_store: used documents would be counted as success whereas failure_store: failed and failure_store: not_enabled would be counted as failed?

[1pkg]

My idea behind it that the existing SLOs stay untouched, with enabled failure store they should always be 100% good events in theory. Separately a new set of SLOs can be created to track failure store error rate, and the number of documents indexed to FS vs all documents.

[simitt]

sounds good, thanks

[simitt]

The already reported stats are not very consistent (some of them contain the term docs, others don't - e.g. Indexed vs RetriedDocs, FailedDocs).
However, when reading FailureStore.Used, FailureStore.Failed and FailureStore.NotEnabled, it's not necessarily clear from the naming whether this would count the number of documents or batch requests. Is it too verbose to add Docs?

[1pkg]

I see no problem adding docs to the names. What makes more sense FailureStoreDocs.Used, FailureStoreDocs.Failred, FailureStoreDocs.NotEnabled or alternatively FailureStore.UsedDocs, FailureStore.FailredDocs, FailureStore.NotEnabledDocs?

[simitt]

I'm slightly in favour of FailureStoreDocs.Used, but no strong opinion.