Fleet server / Elastic agent: "output not supported" using Logstash output on 8.6

[ceeeekay]

Hi there,

I posted this over at discuss.elastic.co but didn't get any response. Apologies for skirting the issue rules but 8.6 is currently unusable. I'm reasonably sure it's a bug though, due to the issue occurring when upgrading from 8.5 to 8.6. Downgrading to 8.5 again resolves the issue.

• Versions: 8.6.0 -> 8.6.2
• Operating System: Ubuntu 22.04.2
• Package: .deb
• Discuss Forum URL: https://discuss.elastic.co/t/elastic-agent-output-not-supported-using-logstash-output-on-8-6/326010
• Steps to Reproduce:
  • Configure Logstash TLS output with 8.5.3 agents and server, and observe agents coming online in the Fleet Agents interface.
  • Upgrade to any 8.6 version and observe all agents go offline.
  • Alternatively, upgrade agents to 8.6 while using Elasticsearch output, then switch to Logstash output and observe the agents go offline.

• elastic-agent status on 8.6 agents shows DEGRADED status, and FAILED / output not supported for an unnamed(?) component, *
• Fleet server is not listening on 8220 when in this failed state.
• Re-enrolling Fleet servers doesn't help. They appear as new agent instances but never check in.

---

Discuss post, edited with new info:

I have a previously working^* Fleet/Agent config with 8.5.3 and Logstash output configured.

^{* aside from #1790 but I have a workaround for this.}

When upgraded to 8.6.x, when using the Logstash output, agents will appear offline in the Fleet Agent interface.

# elastic-agent version
Binary: 8.6.2 (build: 913c02bea9b13dec4d5c5f3057b5b397344e3298 at 2023-02-13 16:51:45 +0000 UTC)
Daemon: 8.6.2 (build: 913c02bea9b13dec4d5c5f3057b5b397344e3298 at 2023-02-13 16:51:45 +0000 UTC)

# elastic-agent status
State: DEGRADED
Message: 1 or more components/units in a failed state
Components:
  * beat/metrics    (HEALTHY)
                    Healthy: communicating with pid '80217'
  * log             (HEALTHY)
                    Healthy: communicating with pid '80185'
  * system/metrics  (HEALTHY)
                    Healthy: communicating with pid '80192'
  *                 (FAILED)
                    output not supported
  * http/metrics    (HEALTHY)
                    Healthy: communicating with pid '80201'
  * filestream      (HEALTHY)
                    Healthy: communicating with pid '80208'

Fleet server logs:
{"log.level":"error","@timestamp":"2023-02-24T11:29:45.073+1300","log.origin":{"file.name":"coordinator/coordinator.go","file.line":857},"message":"Spawned new component fleet-server-default: output not supported","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"FAILED"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-02-24T11:29:45.073+1300","log.origin":{"file.name":"coordinator/coordinator.go","file.line":857},"message":"Spawned new unit fleet-server-default-fleet-server-fleet_server-649981c2-40f8-4e21-97f7-e3ccbd19939e: output not supported","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"FAILED"},"unit":{"id":"fleet-server-default-fleet-server-fleet_server-649981c2-40f8-4e21-97f7-e3ccbd19939e","type":"input","state":"FAILED"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-02-24T11:29:45.073+1300","log.origin":{"file.name":"coordinator/coordinator.go","file.line":857},"message":"Spawned new unit fleet-server-default: output not supported","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"FAILED"},"unit":{"id":"fleet-server-default","type":"output","state":"FAILED"},"ecs.version":"1.6.0"}

Logstash, Elasticsearch and Kibana are all 8.6.2.

When using the Elasticsearch output agents will connect, however this is not a solution for me as ES is only reachable by a small subset of my agents.

It's 100% unusable in this state as all agents go offline, and I've had to downgrade the agents to 8.5.3 just to get going again.

[cmacknz]

What is the output of elastic-agent status --output=json when this happens? It should show exactly what is failing without needing to look at the logs.

In this case the problem is the fleet-server itself doesn't support any output but Elasticsearch. This is intentional and has always been the case, fleet-server needs an Elasticsearch output to function. 8.6.x is likely just strictly enforcing this because of some changes we made under the hood.

elastic-agent/specs/fleet-server.spec.yml

Lines 13 to 14 in 98b8fe7

	outputs:
	- elasticsearch

Can you share the agent policy you are using? Fleet server must somehow be getting associated with a Logstash output.

[ceeeekay]

the fleet-server itself doesn't support any output but Elasticsearch

That makes sense. I have no problem with Fleet servers going direct to Elasticsearch - they're permitted to on the network side and were set up that way intentionally. As long as I can also have the regular Agents go via Logstash, but I'm not sure I can configure it that way myself under a basic license - it's certainly not allowed via the Fleet interface, i.e., there's just one output choice for all agents.

Gist of elastic-agent status --output=json

# cat fleet-server.spec.yml
version: 2
inputs:
  - name: fleet-server
    description: "Fleet Server"
    platforms:
      - linux/amd64
      - linux/arm64
      - darwin/amd64
      - darwin/arm64
      - windows/amd64
      - container/amd64
      - container/arm64
    outputs:
      - elasticsearch
    command:
      args:
        - "--agent-mode"
        - "-E"
        - "logging.level=debug"
        - "-E"
        - "logging.to_stderr=true"

Can you share the agent policy you are using?

The actual Fleet Server policy? It's just the default created by Kibana when I first set it up. Gist here.

The revision is high because I keep flapping between ES and LS outputs to get things going.

Cheers.

[cmacknz]

Thanks. In 8.6 we made a change in the agent that causes Fleet Server to refuse to start if it is associated with anything other than an Elasticsearch output.

This appears to be interacting with the restrictions on output configuration with the basic license, causing your configuration from 8.5.3 to fail to work in 8.6.0.

That Fleet Server is refusing to start in this situation is correct from its perspective, but this configuration with a basic license isn't something we specifically decided to break in 8.6. I'll follow up to see what we want to do about it.

[cmacknz]

elastic/kibana#152234 (comment) is the issue you want to follow.

I believe this may have been working by coincidence in 8.5.x, which is how we unintentionally broke it in 8.6.

[ceeeekay]

Nice one, thanks. Close this issue?

[cmacknz]

Yes, closing this one since the fix is tracked somewhere else.