[All OS's]: Agent goes unhealthy when some other output is set to Default output.

[amolnater-qasource]

Kibana version: 8.2 Snapshot Kibana cloud environment

Host OS: Windows

Build details:
VERSION: 8.2.0 Snapshot
BUILD: 51431
COMMIT: a743498436a863e142592cb535b43f44c448851a
Artifact link: https://artifacts-api.elastic.co/v1/search/8.2-SNAPSHOT

Preconditions:

1. 8.2 Snapshot Kibana cloud environment should be available.
2. Windows agent should be installed.

Integrations:
System and Endpoint Security.

Steps to reproduce:

1. Login to Kibana environment.
2. Navigate to Fleet>Agent Policy tab.
3. Set output as "default" for agent policy.
4. Create a new output under Fleet Settings and set it to Default.
5. Observe within 60 seconds Windows Agent goes Unhealthy.

Expected Result:
Agent should not go unhealthy on changing Default output to some other user created output.

Logs:
logs.zip

Screen Recording:

Windows.Agent.Policy.-.Agent.policies.-.Fleet.-.Elastic.-.Google.Chrome.2022-03-28.12-48-34.mp4

Windows Output:

Note:

• Issue is not observed on Linux agent.

[amolnater-qasource]

@manishgupta-qasource Please review.

[manishgupta-qasource]

Secondary review for this ticket is Done

[ph]

@lykkin might be related to you changes?

[amolnater-qasource]

Hi Team
We have revalidated this issue on latest 8.2 BC-4 Kibana cloud environment and found it reproducible on all OS's.
We have updated the title to avoid any confusions.

• We have set another output under policy settings, and it gets Agents Unhealthy.

Integrations:
System and Endpoint Security

Platforms Validated:

• Windows
• Ubuntu
• Centos
• MAC

Build details:
BUILD: 52005
COMMIT: 9a5003d8cf0062bf24ef64d6712b44823888cc03
Artifact Link: https://staging.elastic.co/8.2.0-3b2b9b86/summary-8.2.0.html

Logs:
elastic-agent-diagnostics-2022-04-21T06-36-28Z-00.zip

Screenshot:

Please let us know if anything else is required from our end.
Thanks

[jlind23]

@lykkin any progress here?

[lykkin]

In the logs we see

{"log.level":"error","@timestamp":"2022-03-28T08:00:31.761Z","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2022-03-28T04:00:31-04:00 - message: Application: endpoint-security--8.2.0-SNAPSHOT[3674c80e-9c57-47c2-9f2e-1e967c53b755]: State changed to FAILED: failed to start connection credentials listener: listen tcp 127.0.0.1:6788: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted. - type: 'ERROR' - sub_type: 'FAILED'","ecs.version":"1.6.0"}

repeated many times. Out of curiosity, if you remove endpoint from the agent does the issue still occur?

Also pinging @elastic/endpoint in case this is a known issue.

[amolnater-qasource]

Hi @lykkin
We have revalidated this issue on latest 8.2 BC-4 and yes we have observed that agent goes to unhealthy state only when Endpoint Security is added to it.
Issue is not reproducible without ES.

Build details:
BUILD: 52005
COMMIT: 9a5003d8cf0062bf24ef64d6712b44823888cc03
Artifact Link: https://staging.elastic.co/8.2.0-3b2b9b86/summary-8.2.0.html

Screenshot:

Logs:
elastic-agent-diagnostics-2022-04-29T11-31-28Z-00.zip
endpoint-000000.zip

Thanks

[ferullo]

Can you share elastic-endpoint.yaml?

[amolnater-qasource]

Hi @ferullo
Please find below elastic-endpoint.yaml from 8.2 BC-4 Endpoint:
elastic-endpoint.zip

Please let us know if anything else is required from our end.
Thanks

[intxgo]

The endpoint log looks fine. Endpoint connects successfully to Agent.

The former log is an Agent log:

{"log.level":"error","@timestamp":"2022-03-28T08:00:31.761Z","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2022-03-28T04:00:31-04:00 - message: Application: endpoint-security--8.2.0-SNAPSHOT[3674c80e-9c57-47c2-9f2e-1e967c53b755]: State changed to FAILED: failed to start connection credentials listener: listen tcp 127.0.0.1:6788: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted. - type: 'ERROR' - sub_type: 'FAILED'","ecs.version":"1.6.0"}

It looks like the Agent is trying to start duplicate TCP server on localhost:6788. This localhost port is used by Endpoint as bootstrap connection port. It's really not related to any stack Output and since it's not possible to run several Endpoints on the same host it would make no sense to use any advanced socket options like SO_REUSEADDR on the TCP server side. Most likely it's a bug in Agent logic.

On a side note, I'm curious what is the purpose of the custom output, how should it work? Is it supposed to alter behavior of Endpoint in any way? The only documentation I was able to find it's rather shallow. Digging deeper from that page, I found the following snippet here

Only a single output may be defined.

[jlind23]

@intxgo the other output may have been logstash in this case but we need to keep the Elasticsearch one as it is the one used for Elastic Agent API key and so on..

[intxgo]

Is it expected to have Endpoint output data (events and alerts) to both Elasticsearch and Logstash at the same time if configured?

[jlind23]

Nope, only one should be used then.

[jlind23]

@ph shouldn't we change something on Fleet-ui/Elastic Agent end to avoid having two default outputs?

[jlind23]

ping @ph

[amolnater-qasource]

Bug Conversion

• New testcase is not required for this scenario as it is covered under Exploratory Testing.

Thanks!

[ph]

@jlind23 This is interesting, I would have expected that is not a possible behavior, because the default is a key in a map and the last would have won. We should validate in Fleet UI, and Fleet-Server we shouldn't trust fleet.

@amolnater-qasource Do you have the generated agent policy yml from fleet, please remove any credentials.

[amolnater-qasource]

Hi @ph
Please find below attached elastic-agent.yml:
elastic-agent.zip

We have revalidated this issue on latest 8.3 Snapshot and found it fixed now.

• Agent remains Healthy on changing Default output to some other user created output.

Build details:
BUILD: 52935
COMMIT: 5d5603a57237d8fe9cf186916c713b9ddddf039d

Screenshots:

Hence we are closing this issue and marking this as QA:Validated.

cc: @jlind23
Thanks