logp: don't write to files by default if running in a container environment

[mauri870]

What does this PR do?

The PR #208 introduced a bug that made the -environment container|systemd flag stop writting to stderr on the default logger. This PR restores the old behavior as well as adding a test that observes that logs are properly written to stderr.

Why is it important?

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have added tests that prove my fix is effective or that my feature works

Related issues

• Fixes Beats do not log to stderr when running in a container or systemd environment beats#41118

[cmacknz]

The elastic-agent container always logs to files in the file system by default, even if logging to stdout+stderr is disabled. These are the logs that are included in diagnostics from the container.

Let's make sure we don't break agent diagnostics when this merges into elastic-agent.

[mauri870]

ACK. This is reverting to the previous behavior. When we silently made this change earlier, it didn’t break elastic-agent, nor was it broken before the change. I don’t believe it's affected at all, but I could be wrong.

[mauri870]

So looking at changes in elastic-agent, there is only two occurrences of logp.DefaultConfig and both use logp.DefaultEnvironment which always gets ToFiles = true.

[belimawr]

I was thinking a bit more about it and I believe the problem is not just because we always set toFiles to true, it's also because DefaultConfig sets an output, so when logp.createLogOutput

elastic-agent-libs/logp/core.go

Lines 250 to 272 in 196dace

	func createLogOutput(cfg Config, enab zapcore.LevelEnabler) (zapcore.Core, error) {
	switch {
	case cfg.toIODiscard:
	return makeDiscardOutput(cfg, enab)
	case cfg.ToStderr:
	return makeStderrOutput(cfg, enab)
	case cfg.ToSyslog:
	return makeSyslogOutput(cfg, enab)
	case cfg.ToEventLog:
	return makeEventLogOutput(cfg, enab)
	case cfg.ToFiles:
	return makeFileOutput(cfg, enab)
	}
	switch cfg.environment {
	case SystemdEnvironment, ContainerEnvironment:
	return makeStderrOutput(cfg, enab)
	case MacOSServiceEnvironment, WindowsServiceEnvironment:
	return makeFileOutput(cfg, enab)
	default:
	return zapcore.NewNopCore(), nil
	}
	}

gets called, the environment is not taken into consideration because an output has already been explicitly chosen.

I believe the systemd case (Filebeat running as a service) is also affected. Installing from the deb should trigger this case, I'm not sure if there is an option to "manually" install Filebeat as a service.

Could you also test the deb/systemd flow to make sure we fixed it in all cases?

[mauri870]

Could you also test the deb/systemd flow to make sure we fixed it in all cases?

Sure, I was able to build a deb package with PACKAGES=deb PLATFORMS=linux/amd64 mage package, I'll spin up a vm to test this case.

[mauri870]

@belimawr From my testing with systemd, filebeat creates /var/log/filebeat/*.ndjson files and journalctl -u filebeat.service does not contain any logs. The proper behavior would be for the logs to go to stderr in the systemd case as well and appear in the journalctl output, right?

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 0d15744

History

• 💔 Build #424 failed 3db5712
• 💚 Build #423 succeeded 399ae3f
• 💚 Build #419 succeeded 3b10333
• 💚 Build #418 succeeded 0f7f948

cc @mauri870

[mauri870]

@belimawr This should be working properly now both for systemd and container environments. I tested the deb package of v8.15.0 and it does not log to syslog by default. I managed to package filebeat (main) with this fix included and it seems to log to syslog properly when running via systemd as well as docker, see attached logs. Please let me know if there is any additional concerns on your end.

root@mauri-ubuntu:/home/azureuser# filebeat version
filebeat version 9.0.0 (amd64), libbeat 9.0.0 [775d26d94a5a1c5a16ed2537d629031e6ef6ee65 built 2024-10-09 12:47:12 +0000 UTC]
root@mauri-ubuntu:/home/azureuser# sudo systemctl status filebeat
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
     Loaded: loaded (/usr/lib/systemd/system/filebeat.service; disabled; preset: enabled)
     Active: active (running) since Wed 2024-10-09 13:02:53 UTC; 1min 8s ago
       Docs: https://www.elastic.co/beats/filebeat
   Main PID: 3602 (filebeat)
      Tasks: 8 (limit: 9459)
     Memory: 22.4M (peak: 22.8M)
        CPU: 84ms
     CGroup: /system.slice/filebeat.service
             └─3602 /usr/share/filebeat/bin/filebeat --environment systemd -c /etc/filebeat/filebeat.yml --path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/filebeat --path.logs /var/log/filebeat

Oct 09 13:02:53 mauri-ubuntu filebeat[3602]: {"log.level":"info","@timestamp":"2024-10-09T13:02:53.261Z","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cfgfile.(*Reloader).Run","file.name":"cfgfile/reload.go","file.l>
Oct 09 13:02:53 mauri-ubuntu filebeat[3602]: {"log.level":"error","@timestamp":"2024-10-09T13:02:53.339Z","log.logger":"add_cloud_metadata","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata>
Oct 09 13:02:53 mauri-ubuntu filebeat[3602]: {"log.level":"error","@timestamp":"2024-10-09T13:02:53.339Z","log.logger":"add_cloud_metadata","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata>
Oct 09 13:02:53 mauri-ubuntu filebeat[3602]: {"log.level":"error","@timestamp":"2024-10-09T13:02:53.339Z","log.logger":"add_cloud_metadata","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata>
Oct 09 13:02:53 mauri-ubuntu filebeat[3602]: {"log.level":"error","@timestamp":"2024-10-09T13:02:53.339Z","log.logger":"add_cloud_metadata","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata>
Oct 09 13:02:53 mauri-ubuntu filebeat[3602]: {"log.level":"error","@timestamp":"2024-10-09T13:02:53.340Z","log.logger":"add_cloud_metadata","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata>
Oct 09 13:02:53 mauri-ubuntu filebeat[3602]: {"log.level":"warn","@timestamp":"2024-10-09T13:02:53.358Z","log.logger":"add_cloud_metadata","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.>
Oct 09 13:02:53 mauri-ubuntu filebeat[3602]: {"log.level":"info","@timestamp":"2024-10-09T13:02:53.358Z","log.logger":"add_cloud_metadata","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.>
Oct 09 13:03:23 mauri-ubuntu filebeat[3602]: {"log.level":"info","@timestamp":"2024-10-09T13:03:23.262Z","log.logger":"monitoring","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logS>
Oct 09 13:03:53 mauri-ubuntu filebeat[3602]: {"log.level":"info","@timestamp":"2024-10-09T13:03:53.262Z","log.logger":"monitoring","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logS>
root@mauri-ubuntu:/home/azureuser#

[belimawr]

@belimawr From my testing with systemd, filebeat creates /var/log/filebeat/*.ndjson files and journalctl -u filebeat.service does not contain any logs. The proper behavior would be for the logs to go to stderr in the systemd case as well and appear in the journalctl output, right?

Yes, I believe this is the correct behaviour: when running under systemd, the logs go to stderr and can be seen using journalctl.

The best approach to be sure is to test an older version 😅

I also saw your last comment, the behaviour looks correct.

[belimawr]

The fact that we have this logic in two different places, bugs me... Because the FF for 8.15.3 was Tue, I'd get this PR in as it is and backport it if we still have time to make into the release. Then create an issue to re-visit it at some point in the future.

I believe that's the correct function to define the default behaviour, createLogOutput should not need the environment switch.

[mauri870]

I also saw your last comment, the behaviour looks correct.

Thanks, I did test with 8.14.3 and the behavior seems consistent with with filebeat main + this fix.