Windows service for Beat does not stop when output is unreachable

[intxgo]

Version

8.13.0

Operating System

Windows

Steps to Reproduce:

1. Install the Beat, e.g. Winlogbeat, on Windows.
2. Configure output to nonexisting URL, for example:

output.elasticsearch:
  hosts: ["non-existing-elasticsearch-output.co:9200"]

4. Make sure the Beat's service is registered with the service manager.
5. Start service, verify that it's working
6. Stop serice

Observed behavior

Stop service never stops the Beat service.

>sc.exe stop winlogbeat

SERVICE_NAME: winlogbeat
        TYPE               : 10  WIN32_OWN_PROCESS
        STATE              : 4  RUNNING
                                (STOPPABLE, NOT_PAUSABLE, ACCEPTS_SHUTDOWN)
        WIN32_EXIT_CODE    : 0  (0x0)
        SERVICE_EXIT_CODE  : 0  (0x0)
        CHECKPOINT         : 0x0
        WAIT_HINT          : 0x0

>sc.exe query winlogbeat

SERVICE_NAME: winlogbeat
        TYPE               : 10  WIN32_OWN_PROCESS
        STATE              : 3  STOP_PENDING
                                (NOT_STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN)
        WIN32_EXIT_CODE    : 0  (0x0)
        SERVICE_EXIT_CODE  : 0  (0x0)
        CHECKPOINT         : 0x0
        WAIT_HINT          : 0x0
        
>TASKLIST /FI "IMAGENAME eq winlogbeat.exe"

Image Name                     PID Session Name        Session#    Mem Usage
========================= ======== ================ =========== ============
winlogbeat.exe                4912 Services                   0     71,020 K

The logs indicate repeated messages like this

{"log.level":"warn","@timestamp":"2024-08-07T17:53:22.057+0200","log.logger":"transport","log.origin":{"function":"github.com/elastic/elastic-agent-libs/transport/httpcommon.(*HTTPTransportSettings).RoundTripper.NetDialer.TestNetDialer.func3","file.name":"transport/tcp.go","file.line":52},"message":"DNS lookup failure \"non-existing-elasticsearch.co\": lookup non-existing-elasticsearch.co: no such host","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-08-07T17:53:22.057+0200","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/elastic-agent-libs/transport/httpcommon.(*HTTPTransportSettings).RoundTripper.LoggingDialer.func1","file.name":"transport/logging.go","file.line":38},"message":"Error dialing lookup non-existing-elasticsearch.co: no such host","service.name":"winlogbeat","network":"tcp","address":"non-existing-elasticsearch.co:9200","ecs.version":"1.6.0"}

Expected behavior

The service stops gracefully

Related

A similar issue, which also might be caused by lost connectivity during system shutdown

• Windows service for Beat does not stop gracefully #38666

[elasticmachine]

Pinging @elastic/sec-windows-platform (Team:Security-Windows Platform)

[cmacknz]

Re-introduced by revert #40705