[logstash] align mappings with metricbeat #3993

klacabane · 2022-08-12T12:00:11Z

Summary

Align the logstash data streams' mappings with the metricbeat ones. The change also defines a logstash service that can be started by running elastic-package service up in the logstash directory.

Testing

First make sure you have elastic-package installed.

When testing we have to install the elasticsearch package as well otherwise the logstash section in Stack Monitoring won't show up.

We can automate the package installation by providing the right fleet configuration to kibana, and we can use elastic-package profiles to do that. Let's download a profile that specifically does that. You can skip this step if you want to install the packages manually

curl https://drive.google.com/uc\?export\=download\&id\=18ZhDYugqGUgxT0zE68DQuI9jOdtcvyri -L -o /tmp/stack_monitoring-logstash.zip && \
unzip -o /tmp/stack_monitoring-logstash.zip -d ~/.elastic-package/profiles && \
rm /tmp/stack_monitoring-logstash.zip

Now cd at the root of the integrations repository and let's build elasticsearch/logstash package, start the stack with the downloaded profile and also start a logstash service with some predefined pipelines. The command may take a moment to complete and will be done once the logstash service is started, this log message should appear: Service is up, please use ctrl+c to take it down

(cd packages/elasticsearch && elastic-package build) && \
(
  cd packages/logstash && elastic-package build && \
  elastic-package stack up -v -d --profile stack_monitoring-logstash --version 8.5.0-SNAPSHOT && \
  elastic-package service up -v
)

load kibana at https://localhost:5601, navigate to Stack Monitoring and inspect logstash section. These two pipelines should show up under the elasticsearch cluster while the standalone-pipeline should appear in the Standalone cluster.
Note: If the Standalone Cluster is not showing up it means that this change [Stack Monitoring] support logstash datastream in standalone query kibana#138689 is not available in 8.5 snapshot yet. You can follow the Testing steps from this PR [Stack Monitoring] support for integration package kibana#138224 to connect a local kibana containing the changes

elasticmachine · 2022-08-12T12:15:10Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-08-16T09:32:05.885+0000
Duration: 17 min 56 sec

Test stats 🧪

Test	Results
Failed	0
Passed	15
Skipped	0
Total	15

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2022-08-12T12:15:21Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`0/0`)	💚
Files	100.0% (`0/0`)	💚 2.879
Classes	100.0% (`0/0`)	💚 2.879
Methods	42.857% (`6/14`)	👎 -47.201
Lines	100.0% (`0/0`)	💚 8.173
Conditionals	100.0% (`0/0`)	💚

matschaffer · 2022-08-15T22:41:09Z

We can automate the package installation by providing the right fleet configuration to kibana, and we can use elastic-package profiles

That's a really interesting step. Nice work figuring out how to package/distribute the profile. It makes me wonder if there should be something a little more "first class" for automating installation (for example elastic-package stack install logstash). But not something I'd consider in-scope for this PR.

matschaffer · 2022-08-15T22:44:07Z

Also scratching my head at what coverage counting tool is flagging -46.357 for a PR that's all configuration as far as I can see.

matschaffer · 2022-08-15T23:32:27Z

My initial attempt to run your instructions didn't give me the agent package, just the filebeat/metricbeat "instructional" (? not sure if right word) integrations.

Trying this:

 ~/elastic/integrations/packages/logstash
❯ elastic-package build; elastic-package stack up -v -d --services package-registry

(I think I'm also warming regional caches this morning, so far 20min in total for all the docker-compose ups 😆 )

Update: I think I broke it

Guess I'll use this as an excuse to try to learn more about how the package registry works...

matschaffer · 2022-08-16T00:30:56Z

Looks like that command used a different profile, so kibana was failing to query to refreshed package service:

❯ docker logs elastic-package-stack_kibana_1
[2022-08-16T00:28:50.000+00:00][ERROR][plugins.fleet] Error connecting to package registry: request to https://package-registry:8080/search?experimental=true&kibana.version=8.5.0 failed, reason: self signed certificate in certificate chain

Trying this now:

❯ elastic-package stack up -v -d --profile stack_monitoring-logstash --services package-registry --version 8.5.0-SNAPSHOT

Update: error is gone, still no logstash package. I'm gonna retry with default profiles and see if that changes anything.

matschaffer · 2022-08-16T01:12:03Z

While waiting for stacks to do stack things I think I found the coverage report https://beats-ci.elastic.co/job/Ingest-manager/job/integrations/job/main/1196/cobertura/logstash/node/

Looks like it's borrowing the cobertura format so Methods here seems to translate to files that get exercised?

matschaffer · 2022-08-16T01:18:51Z

So I ended up needing to switch the package to release: ga but then I realized, maybe the package was actually installed I just can't see it in the UI. I'll try the description steps again and see if that's the case.

matschaffer · 2022-08-16T01:42:06Z

I wasn't able to get the elasticsearch package to show up in the UI, so I enabled internal monitoring and was able to see logstash.

The setup mode N/A seems to indicate that the data is coming in via agent.

To cross check I looked at the health api but I think maybe it doesn't support the new indices yet?

Would be nice if there were an easy way to pull a source-clone kibana into the stack to be able to fix and test the code accordingly. (Update: oh nice, you have that in elastic/kibana#138224)

matschaffer · 2022-08-16T01:58:54Z

Okay, so confusion resolved here. Looks like the profile works, but since they're not ga they only show in "installed integrations" not in the "browse integrations"?

Aside: the ES logo looks broken, so I opened elastic/kibana#138865 to fix

matschaffer

After a bit of head-desking this seems to work. I had a couple of questions, so just commenting for now.

I'd also like to see if I can test CFS & config reloading to get the reload counters or pipeline versioning to do something other than just be blank.

matschaffer · 2022-08-16T01:52:47Z

packages/logstash/_dev/deploy/docker/docker-compose.yml

+version: '2.3'
+services:
+  logstash:
+    image: "docker.elastic.co/logstash/logstash:8.5.0-SNAPSHOT"


Any idea if we can parameterize this to the latest stack snapshot? Thinking something like what we do on the beats repos.

Would that be the master-SNAPSHOT image ?

Worth a try at least :) I did the same in https://github.com/elastic/beats/blob/main/docs/devguide/modules-dev-guide.asciidoc#test

I think my ideal would be some sort of automation like elastic/beats#32720 that explicitly bumps to updated versions. That way we avoid spurious main branch failures on this repo if something is broken in an updated stack version.

Either way I think we can defer that work to a later PR.

I opened up #4013 to follow up on this and added it to our integration meta.

matschaffer · 2022-08-16T02:02:17Z

packages/logstash/data_stream/node_stats/fields/fields.yml

+                          fields:
+                            - name: number_of_elapsed_periods
+                              type: long
+                            - name: time_throttled_nanos


Do these get validated via a test or something? The cgroup graphs are blank when I test it manually, but I think that might be just due to no quota being set. Will see if I can add one.

matschaffer · 2022-08-16T02:05:32Z

Oh, also is there anything I can do with https://github.com/klacabane/metrics-mappings-to-monitoring to cross-check the mapping diffs?

matschaffer · 2022-08-16T02:29:56Z

heh, I ran into another funny "sharp edge" on elastic-package. Looks like I was running an old docker-compose via pip so I removed that figuring v2 might be easier to set cpu quotas with, but that causes the hostnames to change so it's elastic-package-service-logstash-1 not elastic-package-service_logstash_1

matschaffer · 2022-08-16T02:38:36Z

Well, I can't seem to get logstash to return any cgroup stats even with a quota set:

❯ docker exec -it $(docker ps -qlf Name=logstash) curl localhost:9600'/_node/stats/os?pretty'
{
  "host" : "5d112bb6717c",
  "version" : "8.5.0",
  "http_address" : "0.0.0.0:9600",
  "id" : "5630eddf-666b-42ec-8085-54b87bab8c13",
  "name" : "5d112bb6717c",
  "ephemeral_id" : "ca910dd5-bf0d-4165-8c74-3713f9c4a891",
  "status" : "green",
  "snapshot" : true,
  "pipeline" : {
    "workers" : 3,
    "batch_size" : 125,
    "batch_delay" : 50
  },
  "os" : { }
}

❯ docker inspect $(docker ps -qlf Name=logstash) --format '{{json .HostConfig.CpuQuota}}'
50000

matschaffer · 2022-08-16T02:42:13Z

I can see cpu periods/throttling info in /sys so maybe we're just hitting a logstash bug here:

❯ docker exec -it $(docker ps -qf Name=logstash) cat /sys/fs/cgroup/cpu.stat
usage_usec 127507926
user_usec 119128284
system_usec 8379642
nr_periods 39180
nr_throttled 465
throttled_usec 6879286

packages/logstash/_dev/deploy/docker/config/logstash.yml

matschaffer · 2022-08-16T05:19:18Z

Also confirmed standalone using your instructions from elastic/kibana#138224 - thanks for that! Thinking we should probably start an integrations.md dev doc for stack monitoring to capture some of these techniques.

klacabane · 2022-08-16T09:29:47Z

My initial attempt to run your instructions didn't give me the agent package, just the filebeat/metricbeat "instructional" (? not sure if right word) integrations.

When not GA the integrations won't appear in the UI, but you can still find them with the global search bar. if you type elasticsearch/kibana/logstash it should show an integration option. You can also navigate to the integration through the metricbeat tutorial, they have a link to the agent package.

Looks like that command used a different profile, so kibana was failing to query to refreshed package service:

Ah got bit by that several time and added it to the (still local) docs. profile have their own certificates and CAs so the package-registry needs to be started with the --profile as well to be able to connect back.

To cross check I looked at the health api but I think maybe it doesn't support the new indices yet?

Correct, health api does not support metrics-* yet

Well, I can't seem to get logstash to return any cgroup stats even with a quota set

Let me confirm that on my side and I'll create a ticket

Do these get validated via a test or something?

We can implement system tests (https://github.com/elastic/elastic-package/blob/main/docs/howto/system_testing.md) that verifies that the mapped properties are correctly ingested. I wanted to push their implementation a bit later since I expect those to fail and spot properties that are not populated (see cgroup) and increase the scope of the PR. Our testing strategy for now is to rely on Stack Monitoring UI behaving similarly when loaded with agent and metricbeat, then we can add these tests as a mapping testing and optimization ticket, wdyt ?

Co-authored-by: Mat Schaffer <[email protected]>

matschaffer · 2022-08-17T05:41:23Z

if you type elasticsearch/kibana/logstash it should show an integration option. You can also navigate to the integration through the metricbeat tutorial, they have a link to the agent package.

ahhh... big TIL on that one for me. Thanks!

Our testing strategy for now is to rely on Stack Monitoring UI behaving similarly when loaded with agent and metricbeat, then we can add these tests as a mapping testing and optimization ticket, wdyt ?

Makes sense to me. Looks like #4008 has that covered 👍🏻

matschaffer

Good to merge I think. Definitely some outstanding questions, but we can address them as work carries on in the main meta(s).

klacabane added 4 commits August 12, 2022 13:56

ecs fields

a4b4ba3

node data stream fields

8e9ebd4

node_stats data stream fields

b44b810

_dev/deploy configuration

0eb2948

klacabane self-assigned this Aug 12, 2022

klacabane added Integration:logstash Logstash v8.5.0 Team:Infra Monitoring UI - DEPRECATED Label for the Infrastructure Monitoring UI team. - DEPRECATED - Use Team:obs-ux-infra_services labels Aug 12, 2022

klacabane marked this pull request as ready for review August 12, 2022 13:14

klacabane requested a review from a team as a code owner August 12, 2022 13:14

miltonhultgren self-requested a review August 12, 2022 13:18

matschaffer mentioned this pull request Aug 16, 2022

[Stack monitoring] Add data stream patterns to monitoring health endpoint elastic/kibana#138864

Closed

matschaffer reviewed Aug 16, 2022

View reviewed changes

packages/logstash/_dev/deploy/docker/config/logstash.yml Show resolved Hide resolved

miltonhultgren removed their request for review August 16, 2022 05:29

Update packages/logstash/_dev/deploy/docker/config/logstash.yml

565528d

Co-authored-by: Mat Schaffer <[email protected]>

This was referenced Aug 16, 2022

[Stack Monitoring] Add stack_monitoring suffix to metrics-* index pattern elastic/kibana#137904

Merged

[Stack Monitoring] health api support for data ingested from package elastic/kibana#138964

Merged

matschaffer approved these changes Aug 17, 2022

View reviewed changes

matschaffer mentioned this pull request Aug 17, 2022

[Stack Monitoring] Support automatic update of stack version dependencies #4013

Closed

klacabane merged commit 6422887 into elastic:main Aug 17, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[logstash] align mappings with metricbeat #3993

[logstash] align mappings with metricbeat #3993

klacabane commented Aug 12, 2022 •

edited

Loading

elasticmachine commented Aug 12, 2022 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Aug 12, 2022 •

edited

Loading

matschaffer commented Aug 15, 2022

matschaffer commented Aug 15, 2022 •

edited

Loading

matschaffer commented Aug 15, 2022 •

edited

Loading

matschaffer commented Aug 16, 2022 •

edited

Loading

matschaffer commented Aug 16, 2022 •

edited

Loading

matschaffer commented Aug 16, 2022

matschaffer commented Aug 16, 2022 •

edited

Loading

matschaffer commented Aug 16, 2022

matschaffer left a comment

matschaffer Aug 16, 2022

klacabane Aug 16, 2022

matschaffer Aug 17, 2022

matschaffer Aug 17, 2022

matschaffer Aug 16, 2022

matschaffer commented Aug 16, 2022

matschaffer commented Aug 16, 2022

matschaffer commented Aug 16, 2022

matschaffer commented Aug 16, 2022

matschaffer commented Aug 16, 2022

klacabane commented Aug 16, 2022 •

edited

Loading

matschaffer commented Aug 17, 2022

matschaffer left a comment

[logstash] align mappings with metricbeat #3993

[logstash] align mappings with metricbeat #3993

Conversation

klacabane commented Aug 12, 2022 • edited Loading

Summary

Testing

elasticmachine commented Aug 12, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Aug 12, 2022 • edited Loading

🌐 Coverage report

matschaffer commented Aug 15, 2022

matschaffer commented Aug 15, 2022 • edited Loading

matschaffer commented Aug 15, 2022 • edited Loading

matschaffer commented Aug 16, 2022 • edited Loading

matschaffer commented Aug 16, 2022 • edited Loading

matschaffer commented Aug 16, 2022

matschaffer commented Aug 16, 2022 • edited Loading

matschaffer commented Aug 16, 2022

matschaffer left a comment

Choose a reason for hiding this comment

matschaffer Aug 16, 2022

Choose a reason for hiding this comment

klacabane Aug 16, 2022

Choose a reason for hiding this comment

matschaffer Aug 17, 2022

Choose a reason for hiding this comment

matschaffer Aug 17, 2022

Choose a reason for hiding this comment

matschaffer Aug 16, 2022

Choose a reason for hiding this comment

matschaffer commented Aug 16, 2022

matschaffer commented Aug 16, 2022

matschaffer commented Aug 16, 2022

matschaffer commented Aug 16, 2022

matschaffer commented Aug 16, 2022

klacabane commented Aug 16, 2022 • edited Loading

matschaffer commented Aug 17, 2022

matschaffer left a comment

Choose a reason for hiding this comment

klacabane commented Aug 12, 2022 •

edited

Loading

elasticmachine commented Aug 12, 2022 •

edited

Loading

elasticmachine commented Aug 12, 2022 •

edited

Loading

matschaffer commented Aug 15, 2022 •

edited

Loading

matschaffer commented Aug 15, 2022 •

edited

Loading

matschaffer commented Aug 16, 2022 •

edited

Loading

matschaffer commented Aug 16, 2022 •

edited

Loading

matschaffer commented Aug 16, 2022 •

edited

Loading

klacabane commented Aug 16, 2022 •

edited

Loading