[Elastic Agent] Issue when running the snapshot on macOS

[ph]

2020-04-23T09:31:05-07:00 ERROR	fleet_gateway.go:143	failed to dispatch actions, error: operator: failed to execute step sc-run, error: operation-start: mkdir /var/log/elastic-agent: permission denied: operation-start: mkdir /var/log/elastic-agent: permission denied
	operator: failed to execute step sc-run, error: operation-start: mkdir /var/log/elastic-agent: permission denied: operation-start: mkdir /var/log/elastic-agent: permission denied

This is related to the default path for logging

cc @michalpristas

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[michalpristas]

seems like agent is not run as root and as it performs operations not only related to FS (chown, chmod) but also other ones which might need root permissions i would close this.

I created PR here #17960 but this will not solve issues with root required operations.

Why this happens now is that we let beat log wherever it wants unless we have monitoring configured. this is not possible now as monitoring setting can change on the fly so we configure logging paths at start always

[mtojek]

Unfortunately I faced this issue:

2020-04-24T10:32:49+02:00 ERROR	reporter.go:47	2020-04-24T10:32:49+02:00: type: 'ERROR': sub_type: 'CONFIG' message: Application: filebeat[eec29730-8605-11ea-b266-375c3f1ddf72]: operation-start: chown /var/log/elastic-agent/default: operation not permitted
2020-04-24T10:32:49+02:00 DEBUG	action_dispatcher.go:93	Failed to dispatch action 'action_id: f1351dd0-8605-11ea-b266-375c3f1ddf72, type: CONFIG_CHANGE', error: operator: failed to execute step sc-run, error: operation-start: chown /var/log/elastic-agent/default: operation not permitted: operation-start: chown /var/log/elastic-agent/default: operation not permitted
	operator: failed to execute step sc-run, error: operation-start: chown /var/log/elastic-agent/default: operation not permitted: operation-start: chown /var/log/elastic-agent/default: operation not permitted
2020-04-24T10:32:49+02:00 ERROR	fleet_gateway.go:143	failed to dispatch actions, error: operator: failed to execute step sc-run, error: operation-start: chown /var/log/elastic-agent/default: operation not permitted: operation-start: chown /var/log/elastic-agent/default: operation not permitted
	operator: failed to execute step sc-run, error: operation-start: chown /var/log/elastic-agent/default: operation not permitted: operation-start: chown /var/log/elastic-agent/default: operation not permitted

Running as root causes that the agent hangs on:

2020-04-24T10:35:57+02:00 INFO	reporter.go:51	2020-04-24T10:35:57+02:00: type: 'STATE': sub_type: 'STARTING' message: Application: filebeat[eec29730-8605-11ea-b266-375c3f1ddf72]: State change: STARTING
2020-04-24T10:35:57+02:00 INFO	process.go:73	address assigned to the process '/Users/marcin.tojek/go/src/github.com/elastic/beats/x-pack/elastic-agent/build/distributions/elastic-agent-8.0.0-darwin-x86_64/data/install/filebeat-8.0.0-darwin-x86_64/filebeat': '127.0.0.1:11801'

I worked on the nginx integration, upgraded elastic-agent and I can't start it, so I'm blocked currently. /cc @ruflin

[michalpristas]

this can take a while how long does it hang there

[mtojek]

Let me rerun this - well, previously it took zero-time to go through this, hence reporting.

EDIT:

after 2 minutes:

2020-04-24T13:25:42+02:00 INFO	operation_fetch.go:65	filebeat.8.0.0 already exists in /Users/marcin.tojek/go/src/github.com/elastic/beats/x-pack/elastic-agent/build/distributions/elastic-agent-8.0.0-darwin-x86_64/data/downloads/filebeat-8.0.0-darwin-x86_64.tar.gz. Skipping operation operation-fetch
2020-04-24T13:25:42+02:00 INFO	operator.go:226	operation 'operation-fetch' skipped for filebeat.8.0.0
2020-04-24T13:25:42+02:00 INFO	operator.go:226	operation 'operation-verify' skipped for filebeat.8.0.0
2020-04-24T13:25:42+02:00 INFO	operator.go:226	operation 'operation-install' skipped for filebeat.8.0.0
2020-04-24T13:25:42+02:00 DEBUG	operator.go:230	running operation 'operation-start' for filebeat.8.0.0
2020-04-24T13:25:42+02:00 INFO	reporter.go:51	2020-04-24T13:25:42+02:00: type: 'STATE': sub_type: 'STARTING' message: Application: filebeat[4a078890-861e-11ea-888d-398552fd6075]: State change: STARTING
2020-04-24T13:25:43+02:00 INFO	process.go:73	address assigned to the process '/Users/marcin.tojek/go/src/github.com/elastic/beats/x-pack/elastic-agent/build/distributions/elastic-agent-8.0.0-darwin-x86_64/data/install/filebeat-8.0.0-darwin-x86_64/filebeat': '127.0.0.1:16053'
2020-04-24T13:27:41+02:00 INFO	reporter.go:51	2020-04-24T13:27:41+02:00: type: 'STATE': sub_type: 'IN_PROGRESS' message: Application: filebeat[4a078890-861e-11ea-888d-398552fd6075]: State change: IN_PROGRESS
2020-04-24T13:27:41+02:00 DEBUG	operator.go:230	running operation 'operation-config' for filebeat.8.0.0
2020-04-24T13:28:20+02:00 ERROR	reporter.go:47	2020-04-24T13:28:20+02:00: type: 'ERROR': sub_type: 'CONFIG' message: Application: filebeat[4a078890-861e-11ea-888d-398552fd6075]: application 'filebeat--8.0.0' crashed: /Users/marcin.tojek/go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/core/plugin/app/app.go[206]: unknown error
2020-04-24T13:28:20+02:00 INFO	process.go:73	address assigned to the process '/Users/marcin.tojek/go/src/github.com/elastic/beats/x-pack/elastic-agent/build/distributions/elastic-agent-8.0.0-darwin-x86_64/data/install/filebeat-8.0.0-darwin-x86_64/filebeat': '127.0.0.1:11026'

[michalpristas]

can you also check /var/log/elastic-agent/default/filebeat to see anything what does not fit? cannot reproduce here so far

[mtojek]

I'm rebuilding the agent right now and will see.