Telemetry missing when Logstash is monitored exclusively by Metricbeat

[yaauie]

Kibana version: Any 7.17-8.x (including 7.17.19 and 8.13.0)

Elasticsearch version: Any 7.17-8.x (including 7.17.19 and 8.13.0)

Server OS version: ANY

Browser version: ANY

Browser OS version: ANY

Original install method (e.g. download page, yum, from source, etc.): ANY

Describe the bug:

When a Logstash has monitoring data that has exclusively been sent via Metricbeat, Monitoring Telemetry fails to send telemetry data.

Steps to reproduce:

1. Set up Elasticsearch and Kibana with security and stack monitoring as usual, with telemetry enabled, making note of its cluster UUID
2. Set up and run Logstash without monitoring itself

• ${LOGSTASH_HOME}/config/logstash.yml should have:

  monitoring.enabled: false
  monitoring.cluster_uuid: "MONITORING_CLUSTER_UUID_HERE"
  

• Execute a simple heartbeat pipeline and leave it running:
  ${LOGSTASH_HOME}/bin/logstash --config.string="input { heartbeat {} } output { stdout { codec => dots } }"

3. Set up Metricbeat to observe only Logstash

• Create a role beats_system_expanded that has additional permissions necessary to push logstash monitoring data:
  • cluster: monitor, cluster:admin/xpack/monitoring/bulk, manage_ilm, manage_index_templates (same as beats_system)
  • index .monitoring-beats-*: create_index, create (same as beats_system)
  • index .monitoring-logstash-*: create_index, create
• Create a user beats_system_expanded with role beats_system_expanded and record its password
• Configure metricbeat:
  • configure the output to send monitoring events to Elasticsearch using our beats_system_expanded user:

    output.elasticsearch:
     # Array of hosts to connect to.
     hosts: ["localhost:9200"]
    
     # Performance preset - one of "balanced", "throughput", "scale",
     # "latency", or "custom".
     preset: balanced
    
     # Protocol - either `http` (default) or `https`.
     protocol: "https"
    
     # Authentication credentials - either API key or username/password.
     #api_key: "id:api_key"
     username: "beats_system_expanded"
     password: "${BEATS_SYSTEM_EXPANDED_PASSWORD}"
    
     ssl.ca_trusted_fingerprint: "${ELASTIC_CA_FINGERPRINT}"

  • configure it to not monitor itself (but include the cluster UUID for good measure):

    # Set to true to enable the monitoring reporter.
    monitoring.enabled: false
    
    # Sets the UUID of the Elasticsearch cluster under which monitoring data for this
    # Metricbeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
    # is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
    monitoring.cluster_uuid: "MONITORING_CLUSTER_UUID_HERE"

  • disable the system module by renaming ${METRICBEAT_HOME}/modules.d/system.yml to ${METRICBEAT_HOME}/modules.d/system.yml.disabled
  • enable logstash monitoring by renaming ${METRICBEAT_HOME}/modules.d/logstash-xpack.yml.disabled to ${METRICBEAT_HOME}/modules.d/logstash-xpack.yml, ensuring that the module's config points to localhost:9600 (where Logstash's API is) and has xpack.enabled: true

    # Module: logstash
    # Docs: https://www.elastic.co/guide/en/beats/metricbeat/main/metricbeat-module-logstash.html
    
    - module: logstash
      xpack.enabled: true
      metricsets: ["node", "node_stats"]
      period: 10s
      hosts: ["127.0.0.1:9600"]
      #username: "user"
      #password: "secret"

  • start metricbeat with console logging enabled: (cd "${METRICBEAT_HOME}"; ./metricbeat -e)

4. In Kibana, observe the monitoring data for Logstash is making its way to the correct indices, ensuring at least one result for each of:

POST /.monitoring-logstash*/_search
{ "query": { "term": {"metricset.name":"node"} } }

POST /.monitoring-logstash*/_search
{ "query": { "term": {"metricset.name":"node_stats"} } }

5. Leave it running for at least 24h to give the Telemetry collector a chance to run

Expected behavior:

The data pushed to telemetry for this cluster ID should have Logstash-related telemetry

Any additional context:

In #122345 Stack Monitoring was extended to allow monitoring from standalone metricbeat, but Telemetry Collection works by reaching into the same monitoring indices and made no matching changes.

• Logstash stats are stored in a logstash.node.stats namespace via metricbeat, and in legacy collection a similar structure was available in a logstash_stats namespace
  • in Telemetry collector, logstash stats only accommodates the logstash_stats namespace.
  • the query also matches docs that have metricset.name of stats (not node_stats), but still includes a collapse on a field that exists only in the logstash_stats namespace.
• Similarly, Logstash state is stored in a logstash.node namespace via metricbeat, while legacy collection had a similar structure in logstash_state.
  • in Telemetry collector, logstash state only accommodates the logstash_state namespace.
  • the query includes a collapse on a field that only exists in the logstash_state namespace.

I have attempted to remedy this by adding the fields to the filter_path directives (and normalizing the results to the old shape), but the existing use of collapse on a legacy-only field greatly expands the scope of the fix. I attempted to add a runtime field to collapse on, but discovered that is not supported (elastic/elasticsearch#66459).

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[afharo]

It might be addressed by #182304