Force ECS and JSON logging for libbeat/logp #28573

michel-laterman · 2021-10-20T19:42:09Z

What does this PR do?

Force the logp package to act as if ECS and JSON has been enabled. Attach the
service.name field to entries only when Config.Beat is not empty.
Remove the logp.Config.ECSEnabled field. Log entries will now be
JSON formatted by default.

Why is it important?

All Elastic stack components will produce ECS conforming logs in 8.0 by default.
This change forces libbeat, and all components using libbeat to produce these logs (including all beats, and the elastic-agent.).

Checklist

My code follows the style guidelines of this project
[] ~~I have commented my code, particularly in hard-to-understand areas~~
I have made corresponding changes to the documentation
- libbeat deprecation (v7.16): Mark logging.ecs and json as deprecated #28772
- elastic-agent deprecation (v7.16): mark logging.ecs and logging.json as deprecated observability-docs#1206
- elastic-agent removal (v8.0): remove agent.logging.ecs and json observability-docs#1208
I have made corresponding change to the default configuration files
[] ~~I have added tests that prove my fix is effective or that my feature works~~
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Related issues

Relates [Libbeat] ECS Logging #15544

Logs

Example of logs from metricbeat 7.15.1 (before ECS is forced)

2021-10-21T07:57:35.938-0700    INFO    [beat]  instance/beat.go:1023   Build info      {"system_info": {"build": {"commit": "5ae799cb1c3c490c9a27b14cb463dc23696bc7d3", "libbeat": "7.15.1", "time": "2021-10-07T22:19:05.000Z", "version": "7.15.1"}}}
2021-10-21T07:57:35.938-0700    INFO    [beat]  instance/beat.go:1026   Go runtime info {"system_info": {"go": {"os":"darwin","arch":"amd64","max_procs":16,"version":"go1.16.6"}}}
...
2021-10-20T14:56:18.190-0700    INFO    [monitoring]    map[file.line:184 file.name:log/log.go] Non-zero metrics in the last 30s        {"service.name": "metricbeat", "monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":4797,"time":{"ms":423}},"total":{"ticks":17312,"time":{"ms":1509},"value":17312},"user":{"ticks":12515,"time":{"ms":1086}}},"info":{"ephemeral_id":"7a615445-2568-476f-a7fa-8a8ee326240e","uptime":{"ms":333127},"version":"8.0.0"},"memstats":{"gc_next":24007776,"memory_alloc":12986584,"memory_total":11498190104,"rss":127217664},"runtime":{"goroutines":84}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"acked":98,"active":0,"batches":9,"total":98},"read":{"bytes":5661},"write":{"bytes":178581}},"pipeline":{"clients":10,"events":{"active":0,"filtered":1,"published":98,"total":99},"queue":{"acked":98}}},"metricbeat":{"system":{"cpu":{"events":3,"success":3},"filesystem":{"events":8,"success":8},"fsstat":{"events":1,"success":1},"load":{"events":3,"success":3},"memory":{"events":3,"success":3},"network":{"events":51,"success":51},"process":{"events":24,"success":24},"process_summary":{"events":3,"success":3},"socket_summary":{"events":3,"success":3}}},"system":{"load":{"1":4.3774,"15":2.7471,"5":3.064,"norm":{"1":0.2736,"15":0.1717,"5":0.1915}}}}, "ecs.version": "1.6.0"}}

Example of logs from metricbeat after ECS is forced on, and JSON is enabled by default an 8.0.0-SNAPSHOT:

{"log.level":"info","@timestamp":"2021-10-25T10:19:39.196-0700","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1026},"message":"Build info","service.name":"metricbeat","system_info":{"build":{"commit":"99b18da15b6d63b1b7eeb4b0658215c8823a500c","libbeat":"8.0.0","time":"2021-10-25T17:05:03.000Z","version":"8.0.0"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-10-25T10:19:39.196-0700","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1029},"message":"Go runtime info","service.name":"metricbeat","system_info":{"go":{"os":"darwin","arch":"amd64","max_procs":16,"version":"go1.17.1"},"ecs.version":"1.6.0"}}
...
{"log.level":"info","@timestamp":"2021-10-25T10:20:09.202-0700","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"metricbeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":536,"time":{"ms":536}},"total":{"ticks":2314,"time":{"ms":2314},"value":2314},"user":{"ticks":1778,"time":{"ms":1778}}},"info":{"ephemeral_id":"9f6a5bf8-4a89-421e-94c8-62a99e40e57f","uptime":{"ms":33092},"version":"8.0.0"},"memstats":{"gc_next":26509248,"memory_alloc":19075312,"memory_sys":62735368,"memory_total":1309229840,"rss":112537600},"runtime":{"goroutines":84}},"libbeat":{"config":{"module":{"running":3,"starts":3},"reloads":1,"scans":1},"output":{"events":{"acked":99,"active":0,"batches":7,"total":99},"read":{"bytes":7349},"type":"elasticsearch","write":{"bytes":527073}},"pipeline":{"clients":10,"events":{"active":0,"filtered":1,"published":99,"retry":39,"total":100},"queue":{"acked":99,"max_events":4096}}},"metricbeat":{"system":{"cpu":{"events":3,"success":3},"filesystem":{"events":8,"success":8},"fsstat":{"events":1,"success":1},"load":{"events":3,"success":3},"memory":{"events":3,"success":3},"network":{"events":47,"success":47},"process":{"events":28,"success":28},"process_summary":{"events":3,"success":3},"socket_summary":{"events":3,"success":3},"uptime":{"events":1,"success":1}}},"system":{"cpu":{"cores":16},"load":{"1":3.9546,"15":4.2715,"5":3.7983,"norm":{"1":0.2472,"15":0.267,"5":0.2374}}}},"ecs.version":"1.6.0"}}

Example of metricbeat running under elastic-agent (ECS forced) on an 8.0.0-SNAPSHOT:

{"log.level":"info","@timestamp":"2021-10-25T16:55:11.618Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1026},"message":"Build info","service.name":"metricbeat","system_info":{"build":{"commit":"2168fb7075fb395c9ffc35f0a61b5110c6233a23","libbeat":"8.
0.0","time":"2021-10-21T21:56:38.000Z","version":"8.0.0"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-10-25T16:55:11.618Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1029},"message":"Go runtime info","service.name":"metricbeat","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":8,"version":"go1.17.1"}
,"ecs.version":"1.6.0"}}
...
{"log.level":"info","@timestamp":"2021-10-25T16:55:41.623Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"metricbeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"cfs":{"period":{"
us":100000}},"id":"/"},"cpuacct":{"id":"/","total":{"ns":11465664917}},"memory":{"id":"/","mem":{"limit":{"bytes":9223372036854771712},"usage":{"bytes":717189120}}}},"cpu":{"system":{"ticks":140,"time":{"ms":147}},"total":{"ticks":360,"time":{"ms":376},"value":360},"user":{"ticks":2
20,"time":{"ms":229}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":18},"info":{"ephemeral_id":"7f29b5fb-a5c8-461f-8c02-06e2df0ff929","uptime":{"ms":30163},"version":"8.0.0"},"memstats":{"gc_next":20633184,"memory_alloc":14704744,"memory_sys":34685960,"memory_total":457
35480,"rss":163028992},"runtime":{"goroutines":88}},"libbeat":{"config":{"module":{"running":11,"starts":11}},"output":{"events":{"acked":59,"active":0,"batches":3,"total":59},"read":{"bytes":3438},"type":"elasticsearch","write":{"bytes":101023}},"pipeline":{"clients":11,"events":{"
active":0,"filtered":1,"published":59,"retry":20,"total":60},"queue":{"acked":59,"max_events":4096}}},"metricbeat":{"system":{"cpu":{"events":3,"success":3},"diskio":{"events":8,"success":8},"filesystem":{"events":2,"success":2},"fsstat":{"events":1,"success":1},"load":{"events":3,"
success":3},"memory":{"events":3,"success":3},"network":{"events":14,"success":14},"process":{"events":17,"success":17},"process_summary":{"events":3,"success":3},"socket_summary":{"events":3,"success":3},"uptime":{"events":3,"success":3}}},"system":{"cpu":{"cores":8},"load":{"1":2.
56,"15":1.43,"5":1.75,"norm":{"1":0.32,"15":0.1788,"5":0.2188}}}},"ecs.version":"1.6.0"}}

Force the logp package to act as if ECS has been enabled. Attach the "service.name" field to entries only when Config.Beat is not empty. Remove the logp.Config.ECSEnabled field.

elasticmachine · 2021-10-20T19:42:11Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

mergify · 2021-10-20T19:42:43Z

This pull request does not have a backport label. Could you fix it @michel-laterman? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

elasticmachine · 2021-10-20T20:59:06Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-11-02T17:52:04.724+0000
Duration: 164 min 26 sec
Commit: 26083fb

Test stats 🧪

Test	Results
Failed	0
Passed	53586
Skipped	5236
Total	58822

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.

jlind23 · 2021-10-21T06:31:18Z

@michel-laterman will this PR completely resolve #15544?

michalpristas · 2021-10-21T07:19:36Z

when testing please check how many ecs.version fields is present in the reported event and if these versions match.
we had an issue with 2 unmatching ecs.version one from beat one from agent. i believe this was fixed but just to be sure

ruflin

Few additional thoughts which are related but not necessarily should block this PR. For 8.0, should we:

Enable json logs by default?
Disable metrics logging by default?

@simitt @axw It would be nice if by 8.0 apm-server and beats are again in line related to logging. What is your take on the above?

ruflin · 2021-10-21T14:38:30Z

@michel-laterman Any chance you could share some example logs from before and after the change?

michel-laterman · 2021-10-21T15:02:33Z

@ruflin, I've added log excerpts to the PR description.

michalpristas · 2021-10-21T15:04:38Z

can you also include those events from before/after?

michel-laterman · 2021-10-21T19:13:43Z

@ruflin, it should be relatively straightforward to force JSON logging + disable logging metrics in this pr as well

axw · 2021-10-25T05:39:57Z

Enable json logs by default?
Disable metrics logging by default?
@simitt @axw It would be nice if by 8.0 apm-server and beats are again in line related to logging. What is your take on the above?

Agreed, I think that would be nice too. JSON+ECS logs seems non-contentious to me. I don't know who is relying on metrics in logs (there are undoubtedly folks who will rely on that), but I feel like defaulting to off is the right thing.

ruflin · 2021-10-25T11:22:50Z

@michel-laterman Is the example you provided above for 8.0 without Elastic Agent correct? I would have expected the logs of metricbeat 8.0 with and without elastic-agent to be the same.

Lets move forward with ECS + JSON in 8.0 as the default and the only option? @axw This will directly impact apm-server as the config options would be removed. It means in 7.16 we should add a deprecation log message to give users a note in 7.16 to already switch over.

For the metrics part, lets do it in a separate PR.

mergify · 2021-10-25T12:33:50Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b logp-ecs upstream/logp-ecs
git merge upstream/master
git push upstream logp-ecs

simitt

No concerns from my side to change to these defaults, as they are aligned with current apm defaults.

But I'm not certain about removing the logging.ecs config option while keeping logging.json. The ecs-logging-go-zap encoder logs everything in json format:

ECS loggers are formatter/encoder plugins for your favorite logging libraries. They make it easy to format your logs into ECS-compatible JSON.

Is there a reason why beats/apm shouldn't always be logging in json format?

simitt · 2021-10-27T11:35:52Z

libbeat/_meta/config/logging.reference.yml.tmpl

@@ -75,8 +75,3 @@ logging.files:

 # Set to true to log messages in JSON format.
 #logging.json: false


Should be updated to true then

ruflin · 2021-10-27T13:18:49Z

@simitt You would remove both config options? I'm on board with this. Like this Beats has only 1 way left on how to write logs which I think is good. The only downside is that json logs are not always too friendly for human consumption but that is something we could solve with tooling.

simitt · 2021-10-27T15:29:55Z

I'm not sure if the non-json logs are used/required for anything or not. My main point was that ecs without json doesn't make sense, so if non-json logs are required, removing the config option for ecs is not great; otherwise, yes, also remove the json config option.

ruflin · 2021-10-29T12:01:35Z

Lets remove both. This will remove also both for APM. @simitt @axw I assume you are on board with this?

axw · 2021-11-01T01:50:50Z

@ruflin yes, I'm on board with removing both.

michel-laterman · 2021-11-01T20:58:21Z

/test

michel-laterman · 2021-11-02T16:28:11Z

/test

michel-laterman · 2021-11-02T17:45:18Z

/package

…in-the-package-binareis * upstream/master: allows disable pod events enrichment with deployment name (elastic#28521) Remove Docker input from Filebeat (elastic#28817) [breaking] Make default_field: false the default for all fields (elastic#28596) Osquerybeat: Improve osquery client connect code (elastic#28848) Add crawler metrics into the stats metricset for Enterprise Search (elastic#28790) Remove the now deprecated appsearch module from metricbeat (elastic#28850) Remove Beat generators (elastic#28816) chore: upload files to Google Storage when they exist (elastic#28836) Revert "chore(ci): disable E2E tests in Beats (elastic#28715)" (elastic#28812) Deprecate generating custom Beats (elastic#28814) [Metricbeat] upgrade flatbuffers to 1.12.1 (elastic#28094) Osquerybeat: Fix restart flags after previously bad config (elastic#28827) Force ECS and JSON logging for libbeat/logp (elastic#28573) Filebeat: Error on startup for unconfigured module (elastic#28818) Deprecate log input in favour of filestream (elastic#28623) Fix some spelling mistakes (elastic#28080)

Force the logp package to act as if ECS and JSON has been enabled. Attach the service.name field to entries only when Config.Beat is not empty. Remove the logp.Config.ECSEnabled field. Log entries will now be JSON formatted by default. (cherry picked from commit 987250d)

Force the logp package to act as if ECS and JSON has been enabled. Attach the service.name field to entries only when Config.Beat is not empty. Remove the logp.Config.ECSEnabled field. Log entries will now be JSON formatted by default. (cherry picked from commit 987250d) Co-authored-by: Michel Laterman <[email protected]>

…in-the-package-binareis * upstream/master: allows disable pod events enrichment with deployment name (elastic#28521) Remove Docker input from Filebeat (elastic#28817) [breaking] Make default_field: false the default for all fields (elastic#28596) Osquerybeat: Improve osquery client connect code (elastic#28848) Add crawler metrics into the stats metricset for Enterprise Search (elastic#28790) Remove the now deprecated appsearch module from metricbeat (elastic#28850) Remove Beat generators (elastic#28816) chore: upload files to Google Storage when they exist (elastic#28836) Revert "chore(ci): disable E2E tests in Beats (elastic#28715)" (elastic#28812) Deprecate generating custom Beats (elastic#28814) [Metricbeat] upgrade flatbuffers to 1.12.1 (elastic#28094) Osquerybeat: Fix restart flags after previously bad config (elastic#28827) Force ECS and JSON logging for libbeat/logp (elastic#28573) Filebeat: Error on startup for unconfigured module (elastic#28818) Deprecate log input in favour of filestream (elastic#28623) Fix some spelling mistakes (elastic#28080)

Force ECS logging for libbeat/logp

71b6f70

Force the logp package to act as if ECS has been enabled. Attach the "service.name" field to entries only when Config.Beat is not empty. Remove the logp.Config.ECSEnabled field.

michel-laterman added enhancement libbeat breaking change Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team labels Oct 20, 2021

michel-laterman requested a review from michalpristas October 20, 2021 19:42

michel-laterman requested review from a team as code owners October 20, 2021 19:42

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Oct 20, 2021

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Oct 20, 2021

Fix ecs tests

2036871

ruflin reviewed Oct 21, 2021

View reviewed changes

Add PR to CHANGELOGs

2168fb7

jlind23 added the v8.0.0 label Oct 25, 2021

jlind23 linked an issue Oct 25, 2021 that may be closed by this pull request

[Libbeat] ECS Logging #15544

Closed

michel-laterman added 2 commits October 25, 2021 09:26

JSON log format by default

1f2af27

Merge remote-tracking branch 'origin/master' into logp-ecs

99b18da

fix metricbeat test

805ec62

simitt reviewed Oct 27, 2021

View reviewed changes

michel-laterman added 3 commits October 29, 2021 11:16

Remove logging JSON config option

7eac477

Remove reference to logging.json from tests

17e64a1

Remove ecs and json from args elastic agent passes to beats

b24ae64

Remove attributes from docs

26083fb

This was referenced Nov 2, 2021

remove agent.logging.ecs and json elastic/observability-docs#1208

Merged

Mark logging.ecs and json as deprecated #28772

Merged

kvch approved these changes Nov 3, 2021

View reviewed changes

michel-laterman changed the title ~~Force ECS logging for libbeat/logp~~ Force ECS and JSON logging for libbeat/logp Nov 3, 2021

michel-laterman mentioned this pull request Nov 4, 2021

[Libbeat] ECS Logging #15544

Closed

michel-laterman merged commit 987250d into elastic:master Nov 4, 2021

michel-laterman deleted the logp-ecs branch November 4, 2021 20:32

michel-laterman added the backport-v8.0.0 Automated backport with mergify label Nov 10, 2021

mergify bot removed the backport-skip Skip notification from the automated backport with mergify label Nov 10, 2021

mergify bot mentioned this pull request Nov 10, 2021

[8.0](backport #28573) Force ECS and JSON logging for libbeat/logp #28918

Merged

axw mentioned this pull request Nov 15, 2021

Stop overriding logging.{ecs,json} defaults elastic/apm-server#6613

Merged

1 task

jlind23 mentioned this pull request Oct 29, 2021

Fleet managed: enable debug mode elastic/elastic-agent#143

Closed

faec mentioned this pull request Apr 4, 2022

[ECS logging] Enable by default #23871

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Force ECS and JSON logging for libbeat/logp #28573

Force ECS and JSON logging for libbeat/logp #28573

michel-laterman commented Oct 20, 2021 •

edited

Loading

elasticmachine commented Oct 20, 2021

mergify bot commented Oct 20, 2021

elasticmachine commented Oct 20, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

jlind23 commented Oct 21, 2021

michalpristas commented Oct 21, 2021 •

edited

Loading

ruflin left a comment

ruflin commented Oct 21, 2021

michel-laterman commented Oct 21, 2021

michalpristas commented Oct 21, 2021

michel-laterman commented Oct 21, 2021

axw commented Oct 25, 2021

ruflin commented Oct 25, 2021 •

edited

Loading

mergify bot commented Oct 25, 2021

simitt left a comment •

edited

Loading

simitt Oct 27, 2021

ruflin commented Oct 27, 2021

simitt commented Oct 27, 2021

ruflin commented Oct 29, 2021

axw commented Nov 1, 2021

michel-laterman commented Nov 1, 2021

michel-laterman commented Nov 2, 2021

michel-laterman commented Nov 2, 2021

		@@ -75,8 +75,3 @@ logging.files:

		# Set to true to log messages in JSON format.
		#logging.json: false

Force ECS and JSON logging for libbeat/logp #28573

Force ECS and JSON logging for libbeat/logp #28573

Conversation

michel-laterman commented Oct 20, 2021 • edited Loading

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Related issues

Logs

elasticmachine commented Oct 20, 2021

mergify bot commented Oct 20, 2021

elasticmachine commented Oct 20, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

jlind23 commented Oct 21, 2021

michalpristas commented Oct 21, 2021 • edited Loading

ruflin left a comment

Choose a reason for hiding this comment

ruflin commented Oct 21, 2021

michel-laterman commented Oct 21, 2021

michalpristas commented Oct 21, 2021

michel-laterman commented Oct 21, 2021

axw commented Oct 25, 2021

ruflin commented Oct 25, 2021 • edited Loading

mergify bot commented Oct 25, 2021

simitt left a comment • edited Loading

Choose a reason for hiding this comment

simitt Oct 27, 2021

Choose a reason for hiding this comment

ruflin commented Oct 27, 2021

simitt commented Oct 27, 2021

ruflin commented Oct 29, 2021

axw commented Nov 1, 2021

michel-laterman commented Nov 1, 2021

michel-laterman commented Nov 2, 2021

michel-laterman commented Nov 2, 2021

michel-laterman commented Oct 20, 2021 •

edited

Loading

elasticmachine commented Oct 20, 2021 •

edited by jenkins-beats-ci bot

Loading

michalpristas commented Oct 21, 2021 •

edited

Loading

ruflin commented Oct 25, 2021 •

edited

Loading

simitt left a comment •

edited

Loading