Stack monitoring.logstash - 8.0 compatibility

[klacabane]

This ticket tracks the changes required to make the SM logstash section compatible with metricbeat 8.x.

• add elasticsearch.cluster.id to logstash metricsets beats#29625
• stack monitoring: update logstash mappings elasticsearch#82152
• support metricbeat nested logstash.node.stats path in every logstash queries Stack monitoring - logstash 8.0 metricbeat standalone compatibility  #122177

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[klacabane]

For the SM Logstash section, we're looking for stats or logstash_stats metricset names but the metricbeat's logstash module only produces node and node_stats, did the metricset name change since we initially implemented mb support ? @chrisronline @sayden Do you have context on this ?

[klacabane]

With a couple of aliases and queries update to search the appropriate metricsets the UI eventually loads.

There is however a failure in the Pipeline viewer - it relies on a type: nested pipelines object which is defined in an ECS path, with a non-ecs path logstash_stats.pipelines that we actually query. The query returns empty results which is somehow expected since our templates only defines the non-ecs property as {"type": "nested"} without pointing to the path that contains the data. I tried setting an alias to the ECS-path to no avail. It seems that we don't have functional tests covering this view, and loading the test_server with the archived logstash_mb data gives the same results.

@matschaffer Would you have an idea on how to handle aliasing on this nested property, or if there is another way to query that data through the legacy path ?

[matschaffer]

I get a 404 on https://github.com/klacabane/metrics-mappings-to-monitoring/blob/main/templates/logstash-template.json#L276-L278 so I might be missing some context. Have you checked if the view works with 7.x metricbeat monitoring? I wonder if it might already be broken.

I tried setting an alias to the ECS-path to no avail.

What did you try exactly? In general nested type support can be a real pain to work with. If we can't alias into it we might need to loop in ES folks to see if we're just using it wrong or if the alias isn't actually supported.

[klacabane]

Updated the link :) I naively replaced nested with alias and a path pointing to the property carrying the data.

Here's a query that won't return any results because it targets the logstash_stats.pipelines property which, according to the mappings, doesn't define anything besides being nested. Now replace any logstash_stats.pipelines occurrence with logstash.node.stats.pipelines (the path containing the data) and it'll return results given a populated indice.

logstash_stats query

GET .monitoring-logstash-8-*/_search
{
    "sort": {
        "timestamp": {
            "order": "desc",
            "unmapped_type": "long"
        }
    },
    "query": {
        "bool": {
            "filter": [
                {
                    "bool": {
                        "should": [
                            {
                                "term": {
                                    "type": "stats"
                                }
                            },
                            {
                                "term": {
                                    "type": "logstash_stats"
                                }
                            },
                            {
                                "term": {
                                    "type": "node_stats"
                                }
                            },
                            {
                                "term": {
                                    "metricset.name": "stats"
                                }
                            },
                            {
                                "term": {
                                    "metricset.name": "logstash_stats"
                                }
                            },
                            {
                                "term": {
                                    "metricset.name": "node_stats"
                                }
                            }
                        ]
                    }
                },
                {
                    "nested": {
                        "path": "logstash_stats.pipelines",
                        "query": {
                            "bool": {
                                "filter": [
                                    {
                                        "term": {
                                            "logstash_stats.pipelines.id": "main"
                                        }
                                    }
                                ]
                            }
                        }
                    }
                }
            ]
        }
    },
    "aggs": {
        "pipelines": {
            "nested": {
                "path": "logstash_stats.pipelines"
            },
            "aggs": {
                "scoped": {
                    "filter": {
                        "bool": {
                            "filter": [
                                {
                                    "term": {
                                        "logstash_stats.pipelines.id": "main"
                                    }
                                }
                            ]
                        }
                    },
                    "aggs": {
                        "by_pipeline_hash": {
                            "terms": {
                                "field": "logstash_stats.pipelines.hash",
                                "size": 10000,
                                "order": {
                                    "path_to_root>first_seen": "desc"
                                }
                            },
                            "aggs": {
                                "path_to_root": {
                                    "reverse_nested": {},
                                    "aggs": {
                                        "first_seen": {
                                            "min": {
                                                "field": "logstash_stats.timestamp"
                                            }
                                        },
                                        "last_seen": {
                                            "max": {
                                                "field": "logstash_stats.timestamp"
                                            }
                                        }
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}

[matschaffer]

And setting logstash_stats.pipelines as an alias to logstash.node.stats.pipelines doesn't work I take it?

[klacabane]

Update on the nested path - it turns out we can't alias nested properties. It seems that the way to write queries that fetch both legacy logstash_stats.pipelines and metricbeat logstash.node.stats.pipelines is to write two aggregations, one for each path (see pipelines_nested and pipelines_nested_mb) and aggregate the results in server code.

The current issue is that the implementation of these aggregation is incomplete for metricbeat and missing in critical paths. I got a local revision that fixes all this, I'll push the changes as I clean it up here #122177