[Stack Monitoring] Agent fails to publish docs to metrics-elasticsearch.stack_monitoring.enrich index

[crespocarlos]

Summary

While testing elastic/integrations#3929 it was observed that for logstash metrics collection, the Logs UI shows an entry about an error when publishing to metrics-elasticsearch.stack_monitoring.enrich index. Apparently, the agent lacks some privilege

Steps to reproduce

• Follow the steps described here [Stack Monitoring] update packages dataset integrations#4018
• Go to Logs UI

[matschaffer]

I think this points to a missing data stream (for enrich data I guess) in the elasticsearch package to cover enrichment data. Ideally the agent shouldn't have to create anything since the package installation should create all the required data streams.

[crespocarlos]

It happens also with elasticsearch package when Enrich is enabled. The datastream is created, but the error still happens

There may be something else going on

[klacabane]

What's curious here is that we're trying to index the document in the .monitoring-es-8-mb data stream when it should be metrics-elasticsearch.... Maybe the enrich metricset does not allow the agent to take over and overwrite the target data stream ?

[klacabane]

This might be the culprit https://github.com/elastic/beats/blob/main/metricbeat/module/elasticsearch/enrich/data.go#L104-L105

Notice the lack of if isXpack {} guard, which means we'll always write to .monitoring-es-8-mb indice regardless of the xpack configuration. Now maybe agent only overwrite the indice if it is metricbeat-* ? In that case we'll try to index to .monitoring-es-* from agent which is not a supported use case

[klacabane]

The fix is here elastic/beats#33057. I'll work next on verifying whether this also solves the agent issue, but it should me merged even if it's not related

[klacabane]

I was able to test this change locally with guidance from this documentation (internal), within a workspace structured like this:

├── workspace
   ├── beats
   ├── elastic-agent
   ├── integrations

This fixed the issue and enrich documents are now ingested correctly:

enrich document

{
  "_index": ".ds-metrics-elasticsearch.stack_monitoring.enrich-default-2022.09.13-000001",
  "_id": "34KONoMBGj-lllfkoEdC",
  "_version": 1,
  "_score": 0,
  "_source": {
    "agent": {
      "name": "kevins-macbook-pro.home",
      "id": "3e61f1ec-65f8-4695-a453-189633178fdd",
      "ephemeral_id": "396b7f78-ccf8-43c3-b47c-b3753687c1e9",
      "type": "metricbeat",
      "version": "8.5.0"
    },
    "@timestamp": "2022-09-13T11:14:19.398Z",
    "elasticsearch": {
      "cluster": {
        "name": "elasticsearch",
        "id": "ZUGsmOkhS6qTeBLWBWq2yA"
      },
      "node": {
        "id": "Gb5zig0nSF-pnPE80bD8Tg"
      },
      "enrich": {
        "executed_searches": {
          "total": 0
        },
        "remote_requests": {
          "current": 0,
          "total": 0
        },
        "queue": {
          "size": 0
        }
      }
    },
    "ecs": {
      "version": "8.0.0"
    },
    "data_stream": {
      "namespace": "default",
      "type": "metrics",
      "dataset": "elasticsearch.stack_monitoring.enrich"
    },
    "service": {
      "address": "http://localhost:9201",
      "type": "elasticsearch"
    },
    "elastic_agent": {
      "id": "3e61f1ec-65f8-4695-a453-189633178fdd",
      "version": "8.5.0",
      "snapshot": false
    },
    "host": {
      "hostname": "kevins-macbook-pro.home",
      "os": {
        "build": "21G83",
        "kernel": "21.6.0",
        "name": "macOS",
        "type": "macos",
        "family": "darwin",
        "version": "12.5.1",
        "platform": "darwin"
      },
      "ip": [
        "fe80::aede:48ff:fe00:1122",
        "fe80::18bf:b20e:a441:ea49",
        "192.168.1.15",
        "2a01:cb18:2c9:a300:1cb2:66ec:e9b7:e9",
        "2a01:cb18:2c9:a300:ec39:46fd:6d09:5edf",
        "fe80::e436:3aff:fe67:ae41",
        "fe80::e436:3aff:fe67:ae41",
        "fe80::71cb:bc12:406:ecf6",
        "fe80::5313:818f:2712:2ca0",
        "fe80::ce81:b1c:bd2c:69e"
      ],
      "name": "kevins-macbook-pro.home",
      "id": "A9E3D0FF-4D62-5640-8208-87146C5D1E2A",
      "mac": [
        "82-F4-CA-29-54-00",
        "82-F4-CA-29-54-01",
        "82-F4-CA-29-54-04",
        "82-F4-CA-29-54-05",
        "88-66-5A-4D-95-9E",
        "AA-66-5A-4D-95-9E",
        "AC-DE-48-00-11-22",
        "E6-36-3A-67-AE-41"
      ],
      "architecture": "x86_64"
    },
    "metricset": {
      "period": 10000,
      "name": "enrich"
    },
    "event": {
      "duration": 40832489,
      "agent_id_status": "auth_metadata_missing",
      "ingested": "2022-09-13T11:14:20Z",
      "module": "elasticsearch",
      "dataset": "elasticsearch.stack_monitoring.enrich"
    }
  },
  "fields": {
    "elastic_agent.version": [
      "8.5.0"
    ],
    "host.hostname": [
      "kevins-macbook-pro.home"
    ],
    "host.mac": [
      "82-F4-CA-29-54-00",
      "82-F4-CA-29-54-01",
      "82-F4-CA-29-54-04",
      "82-F4-CA-29-54-05",
      "88-66-5A-4D-95-9E",
      "AA-66-5A-4D-95-9E",
      "AC-DE-48-00-11-22",
      "E6-36-3A-67-AE-41"
    ],
    "elasticsearch.enrich.remote_requests.current": [
      0
    ],
    "service.type": [
      "elasticsearch"
    ],
    "host.os.build": [
      "21G83"
    ],
    "host.ip": [
      "fe80::aede:48ff:fe00:1122",
      "fe80::18bf:b20e:a441:ea49",
      "192.168.1.15",
      "2a01:cb18:2c9:a300:1cb2:66ec:e9b7:e9",
      "2a01:cb18:2c9:a300:ec39:46fd:6d09:5edf",
      "fe80::e436:3aff:fe67:ae41",
      "fe80::e436:3aff:fe67:ae41",
      "fe80::71cb:bc12:406:ecf6",
      "fe80::5313:818f:2712:2ca0",
      "fe80::ce81:b1c:bd2c:69e"
    ],
    "agent.type": [
      "metricbeat"
    ],
    "event.module": [
      "elasticsearch"
    ],
    "host.os.version": [
      "12.5.1"
    ],
    "host.os.kernel": [
      "21.6.0"
    ],
    "host.os.name": [
      "macOS"
    ],
    "agent.name": [
      "kevins-macbook-pro.home"
    ],
    "host.name": [
      "kevins-macbook-pro.home"
    ],
    "elastic_agent.snapshot": [
      false
    ],
    "event.agent_id_status": [
      "auth_metadata_missing"
    ],
    "host.id": [
      "A9E3D0FF-4D62-5640-8208-87146C5D1E2A"
    ],
    "timestamp": [
      "2022-09-13T11:14:19.398Z"
    ],
    "elasticsearch.node.id": [
      "Gb5zig0nSF-pnPE80bD8Tg"
    ],
    "elasticsearch.cluster.name": [
      "elasticsearch"
    ],
    "source_node.uuid": [
      "Gb5zig0nSF-pnPE80bD8Tg"
    ],
    "host.os.type": [
      "macos"
    ],
    "elastic_agent.id": [
      "3e61f1ec-65f8-4695-a453-189633178fdd"
    ],
    "data_stream.namespace": [
      "default"
    ],
    "elasticsearch.enrich.executed_searches.total": [
      0
    ],
    "metricset.period": [
      10000
    ],
    "data_stream.type": [
      "metrics"
    ],
    "event.duration": [
      40832489
    ],
    "elasticsearch.cluster.id": [
      "ZUGsmOkhS6qTeBLWBWq2yA"
    ],
    "host.architecture": [
      "x86_64"
    ],
    "metricset.name": [
      "enrich"
    ],
    "event.ingested": [
      "2022-09-13T11:14:20.000Z"
    ],
    "@timestamp": [
      "2022-09-13T11:14:19.398Z"
    ],
    "cluster_uuid": [
      "ZUGsmOkhS6qTeBLWBWq2yA"
    ],
    "agent.id": [
      "3e61f1ec-65f8-4695-a453-189633178fdd"
    ],
    "ecs.version": [
      "8.0.0"
    ],
    "elasticsearch.enrich.remote_requests.total": [
      0
    ],
    "host.os.platform": [
      "darwin"
    ],
    "elasticsearch.enrich.queue.size": [
      0
    ],
    "service.address": [
      "http://localhost:9201"
    ],
    "data_stream.dataset": [
      "elasticsearch.stack_monitoring.enrich"
    ],
    "agent.ephemeral_id": [
      "396b7f78-ccf8-43c3-b47c-b3753687c1e9"
    ],
    "agent.version": [
      "8.5.0"
    ],
    "host.os.family": [
      "darwin"
    ],
    "event.dataset": [
      "elasticsearch.stack_monitoring.enrich"
    ]
  }
}

---

Testing steps:

1. start an elastic stack: cd integrations && elastic-package stack up -v -d --version 8.5.0-SNAPSHOT
2. start the elasticsearch service: cd integrations/packages/elasticsearch && elastic-package service up -v
3. connect to kibana at https://localhost:5601 and create an elasticsearch integration policy that collects data from the service elasticsearch at http://localhost:9201
4. package the elastic-agent: cd elastic-agent && DEV=true PLATFORMS="darwin/amd64" mage package
5. extract the bundle: cd build/distributions/elastic-agent-8.5.0-darwin-x86_64 && tar xzf elastic-agent-8.5.0-darwin-x86_64.tar.gz
6. copy the policy created in 3. to elastic-agent.yml and update the outputs section with:

   outputs:
     default:
       type: elasticsearch
       hosts:
         - 'https://localhost:9200'
       username: 'elastic'
       password: 'changeme'
       ssl.verification_mode: 'none'
   

7. start the agent: ./elastic-agent run
8. verify elasticsearch.stack_monitoring.enrich documents are ingested