Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Agent throws exception when uninstalling on windows #5952

Closed
marc-gr opened this issue Nov 6, 2024 · 13 comments · Fixed by #6257
Closed

Agent throws exception when uninstalling on windows #5952

marc-gr opened this issue Nov 6, 2024 · 13 comments · Fixed by #6257
Assignees
@blakerouse
Labels
bug Something isn't working Team:Elastic-Agent Label for the Agent team Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team

Comments

@marc-gr
Copy link
Contributor

marc-gr commented Nov 6, 2024

For confirmed bugs, please report:

  • Version: 8.16.0 bc3
  • Operating System: windows
  • Steps to Reproduce:

Install and enroll an agent, then uninstall the agent.

On windows, intermittently elastic-agent will throw an exception when uninstalling. Logs in https://gist.github.com/Trinity2019/adddeec9a5ea7314e34c7d443a6bdc2f
@marc-gr marc-gr added the bug Something isn't working label Nov 6, 2024
@rdner rdner added the Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team label Nov 6, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

@rdner rdner added the Team:Elastic-Agent Label for the Agent team label Nov 6, 2024
@cmacknz
Copy link
Member

cmacknz commented Nov 6, 2024

@mauri870 does anything jump out at you in the linked stack trace, it doesn't look like a normal panic, rather some kind of execution violation considering the only error is an invalid program counter: runtime: g 1 gp=0xc00007c000: unknown pc 0x2

@cmacknz
Copy link
Member

cmacknz commented Nov 6, 2024

There is a similar but different error in #3703 which was ultimately closed as not reproducible.

That one gave us the following:

unexpected fault address 0x1f86480
fatal error: fault
[signal 0xc0000005 code=0x1 addr=0x1f86480 pc=0x639c20]

goroutine 6 [running]:
fatal error: unexpected signal during runtime execution
panic during panic
[signal 0xc0000005 code=0x1 addr=0x63a425 pc=0x660b40]

runtime stack:
fatal error: unexpected signal during runtime execution
stack trace unavailable

@ycombinator
Copy link
Contributor

More occurrences:

@blakerouse
Copy link
Contributor

blakerouse commented Nov 15, 2024 •
edited
Loading

I was able to create an integration tests that reproduces this. The test runs install/uninstall in a loop with the Elastic Agent enrolled into Fleet each time. It normally fails on the 6th to 8th loop.

I am testing against 8.16.0 release build.

>>> (windows-amd64-2022-default) Test output (sudo) (stdout): === FAIL: testing/integration TestRepeatedInstallUninstallFleet/TestRepeatedInstallUninstallFleet-6 (5.93s)
[=== ] Attempting to notify Fleet of uninstall  [1s] runtime: pcdata is 39 and 4 locals stack map entries for github.com/elastic/elastic-agent-libs/transport/tlscommon.verifyCertsWithOpts (targetpc=0x8ecf77)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): fatal error: bad symbol table
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): 
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime stack:
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.throw({0x21ccc01?, 0x20a5b7330c8?})
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/panic.go:1023 +0x65 fp=0x86365ff368 sp=0x86365ff338 pc=0x20f8c5
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.(*stkframe).getStackMap(0x86365ff4e0, 0x18?)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/stkframe.go:202 +0x57b fp=0x86365ff440 sp=0x86365ff368 pc=0x22aa5b
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.scanframeworker(0x86365ff4e0, 0x86365ff540, 0xc00007ab68)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgcmark.go:1064 +0x6f fp=0x86365ff4a0 sp=0x86365ff440 pc=0x1f4cef
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.scanstack(0xc0005028c0, 0xc00007ab68)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgcmark.go:900 +0x267 fp=0x86365ff5d0 sp=0x86365ff4a0 pc=0x1f4767
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.markroot.func1()
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgcmark.go:241 +0xb5 fp=0x86365ff620 sp=0x86365ff5d0 pc=0x1f34b5
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.markroot(0xc00007ab68, 0x24, 0x1)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgcmark.go:215 +0x1a8 fp=0x86365ff6c8 sp=0x86365ff620 pc=0x1f3148
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gcDrain(0xc00007ab68, 0x7)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgcmark.go:1200 +0x3d4 fp=0x86365ff730 sp=0x86365ff6c8 pc=0x1f5294
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gcDrainMarkWorkerIdle(...)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgcmark.go:1114
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gcBgMarkWorker.func2()
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:1406 +0x6f fp=0x86365ff780 sp=0x86365ff730 pc=0x1f176f
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.systemstack(0xc000502380)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/asm_amd64.s:509 +0x49 fp=0x86365ff790 sp=0x86365ff780 pc=0x2474c9
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): 
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): goroutine 10 gp=0xc0003e08c0 m=11 mp=0xc000581008 [GC worker (active)]:
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.systemstack_switch()
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/asm_amd64.s:474 +0x8 fp=0xc0003f3f50 sp=0xc0003f3f40 pc=0x247468
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gcBgMarkWorker()
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:1370 +0x1f7 fp=0xc0003f3fe0 sp=0xc0003f3f50 pc=0x1f1437
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.goexit({})
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/asm_amd64.s:1695 +0x1 fp=0xc0003f3fe8 sp=0xc0003f3fe0 pc=0x249481
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): created by runtime.gcBgMarkStartWorkers in goroutine 1
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:1234 +0x1c
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): 
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): goroutine 1 gp=0xc000098000 m=nil [select]:
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gopark(0xc000710528?, 0x4?, 0x28?, 0x3?, 0xc000710448?)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/proc.go:402 +0xce fp=0xc0009062e0 sp=0xc0009062c0 pc=0x21260e
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.selectgo(0xc000906528, 0xc000710440, 0xc000743100?, 0x0, 0x5a027d?, 0x1)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/select.go:327 +0x725 fp=0xc000906400 sp=0xc0009062e0 pc=0x2240e5
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): net/http.(*Transport).getConn(0xc0000ca000, 0xc0007245c0, {{}, 0x0, {0xc000764c80, 0x5}, {0xc00068caa0, 0x49}, 0x0})
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): net/http/transport.go:1406 +0x5a5 fp=0xc000906610 sp=0xc000906400 pc=0x6061e5
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): net/http.(*Transport).roundTrip(0xc0000ca000, 0xc0007290e0)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): net/http/transport.go:595 +0x73a fp=0xc000906838 sp=0xc000906610 pc=0x601fda
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): net/http.(*Transport).RoundTrip(0x2538d18?, 0xc000293ce0?)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): net/http/roundtrip.go:17 +0x13 fp=0xc000906858 sp=0xc000906838 pc=0x5e71d3
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): go.elastic.co/apm/module/apmhttp/v2.(*roundTripper).RoundTrip(0xc000724480, 0xc0007290e0)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): go.elastic.co/apm/module/apmhttp/[email protected]/client.go:87 +0x1ca fp=0xc000906b00 sp=0xc000906858 pc=0x96056a
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): github.com/elastic/elastic-agent/internal/pkg/remote.(*UserAgentRoundTripper).RoundTrip(0xc000718c20, 0xc0007290e0)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): github.com/elastic/elastic-agent/internal/pkg/remote/round_trippers.go:30 +0x122 fp=0xc000906b60 sp=0xc000906b00 pc=0x97d602
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): github.com/elastic/elastic-agent/internal/pkg/fleetapi/client.(*FleetUserAgentRoundTripper).RoundTrip(0x1faac80?, 0xc000745590?)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): github.com/elastic/elastic-agent/internal/pkg/fleetapi/client/round_trippers.go:24 +0x1b fp=0xc000906b80 sp=0xc000906b60 pc=0xb2bb1b
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): github.com/elastic/elastic-agent/internal/pkg/fleetapi/client.(*ElasticApiVersionRoundTripper).RoundTrip(0xc000718c40, 0xc0007290e0)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): github.com/elastic/elastic-agent/internal/pkg/fleetapi/client/round_trippers.go:85 +0xf7 fp=0xc000906be0 sp=0xc000906b80 pc=0xb2bf17
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): github.com/elastic/elastic-agent/internal/pkg/fleetapi/client.(*FleetAuthRoundTripper).RoundTrip(0xc000718c60, 0xc0007290e0)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): github.com/elastic/elastic-agent/internal/pkg/fleetapi/client/round_trippers.go:49 +0x13d fp=0xc000906c88 sp=0xc000906be0 pc=0xb2bc7d
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): net/http.send(0xc000728fc0, {0x2517e00, 0xc000718c60}, {0x126f1c01?, 0x2b5f73?, 0x392b420?})
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): net/http/client.go:259 +0x5e4 fp=0xc000906e68 sp=0xc000906c88 pc=0x5a46c4
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): net/http.(*Client).send(0xc000726590, 0xc000728fc0, {0x21c5c6d?, 0xc?, 0x392b420?})
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): net/http/client.go:180 +0x98 fp=0xc000906ee0 sp=0xc000906e68 pc=0x5a3f78
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): net/http.(*Client).do(0xc000726590, 0xc000728fc0)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): net/http/client.go:724 +0x8dc fp=0xc0009070d0 sp=0xc000906ee0 pc=0x5a629c
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): net/http.(*Client).Do(...)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): net/http/client.go:590
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): github.com/elastic/elastic-agent/internal/pkg/remote.(*Client).Send(0xc0007680b0, {0x2538ca8, 0xc0002b6690}, {0x21b982a, 0x4}, {0xc00068c9b0, 0x45}, 0x0, 0x0, {0x2517480, ...})
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): github.com/elastic/elastic-agent/internal/pkg/remote/client.go:207 +0x828 fp=0xc000907470 sp=0xc0009070d0 pc=0x97bbe8
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): github.com/elastic/elastic-agent/internal/pkg/fleetapi.(*AuditUnenrollCmd).Execute(0xc000711718, {0x2538ca8, 0xc0002b6690}, 0xc0007453e0)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): github.com/elastic/elastic-agent/internal/pkg/fleetapi/audit_unenroll_cmd.go:83 +0x1ec fp=0xc0009075d8 sp=0xc000907470 pc=0xb32d0c
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): github.com/elastic/elastic-agent/internal/pkg/agent/install.notifyFleetAuditUninstall({0x2538c00?, 0x39b48e0?}, 0xc000669000, 0xc00037c780, 0xc0006b00c0, 0xc0006b4c30)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): github.com/elastic/elastic-agent/internal/pkg/agent/install/uninstall.go:173 +0x39a fp=0xc000907810 sp=0xc0009075d8 pc=0xb6ab3a
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): github.com/elastic/elastic-agent/internal/pkg/agent/install.Uninstall({0x2538c00, 0x39b48e0}, {0xc0002dee40, 0x30}, {0xc00017a2e0, 0x1e}, {0x0, 0x0}, 0xc000669000, 0xc00037c780)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): github.com/elastic/elastic-agent/internal/pkg/agent/install/uninstall.go:148 +0x6b3 fp=0xc000907968 sp=0xc000907810 pc=0xb6a3d3
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): github.com/elastic/elastic-agent/internal/pkg/agent/cmd.uninstallCmd(0xc0005366f0, 0xc0003a3508)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): github.com/elastic/elastic-agent/internal/pkg/agent/cmd/uninstall.go:97 +0x585 fp=0xc000907bb8 sp=0xc000907968 pc=0x1ba1ee5
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): github.com/elastic/elastic-agent/internal/pkg/agent/cmd.newUninstallCommandWithArgs.func1(0xc0001aac00?, {0x21b997a?, 0x4?, 0x21b983a?})
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): github.com/elastic/elastic-agent/internal/pkg/agent/cmd/uninstall.go:30 +0x26 fp=0xc000907c38 sp=0xc000907bb8 pc=0x1ba1886
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): github.com/spf13/cobra.(*Command).execute(0xc0003a3508, {0xc000668fb0, 0x1, 0x1})
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): github.com/spf13/[email protected]/command.go:989 +0xab1 fp=0xc000907dc0 sp=0xc000907c38 pc=0x3ced91
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): github.com/spf13/cobra.(*Command).ExecuteC(0xc0003a2308)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): github.com/spf13/[email protected]/command.go:1117 +0x3ff fp=0xc000907e98 sp=0xc000907dc0 pc=0x3cf67f
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): github.com/spf13/cobra.(*Command).Execute(0xc0000b0080?)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): github.com/spf13/[email protected]/command.go:1041 +0x13 fp=0xc000907eb0 sp=0xc000907e98 pc=0x3cf1d3
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): main.main()
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): github.com/elastic/elastic-agent/main.go:32 +0x1ba fp=0xc000907f50 sp=0xc000907eb0 pc=0x1ba727a
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.main()
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/proc.go:271 +0x28b fp=0xc000907fe0 sp=0xc000907f50 pc=0x2121eb
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.goexit({})
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/asm_amd64.s:1695 +0x1 fp=0xc000907fe8 sp=0xc000907fe0 pc=0x249481
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): 
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): goroutine 2 gp=0xc000098700 m=nil [force gc (idle)]:
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/proc.go:402 +0xce fp=0xc00009bfa8 sp=0xc00009bf88 pc=0x21260e
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.goparkunlock(...)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/proc.go:408
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.forcegchelper()
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/proc.go:326 +0xb8 fp=0xc00009bfe0 sp=0xc00009bfa8 pc=0x212478
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.goexit({})
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/asm_amd64.s:1695 +0x1 fp=0xc00009bfe8 sp=0xc00009bfe0 pc=0x249481
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): created by runtime.init.6 in goroutine 1
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/proc.go:314 +0x1a
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): 
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): goroutine 3 gp=0xc000098a80 m=nil [GC sweep wait]:
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/proc.go:402 +0xce fp=0xc00009df80 sp=0xc00009df60 pc=0x21260e
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.goparkunlock(...)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/proc.go:408
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.bgsweep(0xc000058070)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgcsweep.go:318 +0xdf fp=0xc00009dfc8 sp=0xc00009df80 pc=0x1fa93f
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gcenable.gowrap1()
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:203 +0x25 fp=0xc00009dfe0 sp=0xc00009dfc8 pc=0x1ef1e5
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.goexit({})
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/asm_amd64.s:1695 +0x1 fp=0xc00009dfe8 sp=0xc00009dfe0 pc=0x249481
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): created by runtime.gcenable in goroutine 1
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:203 +0x66
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): 
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): goroutine 4 gp=0xc000098c40 m=nil [GC scavenge wait]:
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gopark(0x10000?, 0x2509bf8?, 0x0?, 0x0?, 0x0?)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/proc.go:402 +0xce fp=0xc0000adf78 sp=0xc0000adf58 pc=0x21260e
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.goparkunlock(...)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/proc.go:408
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.(*scavengerState).park(0x392b880)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgcscavenge.go:425 +0x49 fp=0xc0000adfa8 sp=0xc0000adf78 pc=0x1f82c9
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.bgscavenge(0xc000058070)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgcscavenge.go:658 +0x59 fp=0xc0000adfc8 sp=0xc0000adfa8 pc=0x1f8879
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gcenable.gowrap2()
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:204 +0x25 fp=0xc0000adfe0 sp=0xc0000adfc8 pc=0x1ef185
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.goexit({})
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/asm_amd64.s:1695 +0x1 fp=0xc0000adfe8 sp=0xc0000adfe0 pc=0x249481
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): created by runtime.gcenable in goroutine 1
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:204 +0xa5
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): 
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): goroutine 5 gp=0xc000098fc0 m=nil [finalizer wait]:
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gopark(0x20a1624b008?, 0xc000062030?, 0x0?, 0x0?, 0x0?)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/proc.go:402 +0xce fp=0xc00009fe20 sp=0xc00009fe00 pc=0x21260e
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.runfinq()
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mfinal.go:194 +0x107 fp=0xc00009ffe0 sp=0xc00009fe20 pc=0x1ee267
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.goexit({})
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/asm_amd64.s:1695 +0x1 fp=0xc00009ffe8 sp=0xc00009ffe0 pc=0x249481
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): created by runtime.createfing in goroutine 1
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mfinal.go:164 +0x3d
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): 
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): goroutine 18 gp=0xc000104540 m=nil [GC worker (idle)]:
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/proc.go:402 +0xce fp=0xc0000a9f50 sp=0xc0000a9f30 pc=0x21260e
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gcBgMarkWorker()
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:1310 +0xe5 fp=0xc0000a9fe0 sp=0xc0000a9f50 pc=0x1f1325
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.goexit({})
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/asm_amd64.s:1695 +0x1 fp=0xc0000a9fe8 sp=0xc0000a9fe0 pc=0x249481
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): created by runtime.gcBgMarkStartWorkers in goroutine 1
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:1234 +0x1c
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): 
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): goroutine 34 gp=0xc000320000 m=nil [GC worker (idle)]:
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/proc.go:402 +0xce fp=0xc000327f50 sp=0xc000327f30 pc=0x21260e
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gcBgMarkWorker()
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:1310 +0xe5 fp=0xc000327fe0 sp=0xc000327f50 pc=0x1f1325
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.goexit({})
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/asm_amd64.s:1695 +0x1 fp=0xc000327fe8 sp=0xc000327fe0 pc=0x249481
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): created by runtime.gcBgMarkStartWorkers in goroutine 1
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:1234 +0x1c
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): 
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): goroutine 6 gp=0xc0003e0000 m=nil [GC worker (idle)]:
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/proc.go:402 +0xce fp=0xc0000aff50 sp=0xc0000aff30 pc=0x21260e
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gcBgMarkWorker()
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:1310 +0xe5 fp=0xc0000affe0 sp=0xc0000aff50 pc=0x1f1325
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.goexit({})
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/asm_amd64.s:1695 +0x1 fp=0xc0000affe8 sp=0xc0000affe0 pc=0x249481
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): created by runtime.gcBgMarkStartWorkers in goroutine 1
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:1234 +0x1c
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): 
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): goroutine 19 gp=0xc000104700 m=nil [GC worker (idle)]:
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/proc.go:402 +0xce fp=0xc0000abf50 sp=0xc0000abf30 pc=0x21260e
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gcBgMarkWorker()
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:1310 +0xe5 fp=0xc0000abfe0 sp=0xc0000abf50 pc=0x1f1325
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.goexit({})
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/asm_amd64.s:1695 +0x1 fp=0xc0000abfe8 sp=0xc0000abfe0 pc=0x249481
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): created by runtime.gcBgMarkStartWorkers in goroutine 1
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:1234 +0x1c
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): 
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): goroutine 20 gp=0xc0001048c0 m=nil [GC worker (idle)]:
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/proc.go:402 +0xce fp=0xc000323f50 sp=0xc000323f30 pc=0x21260e
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gcBgMarkWorker()
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:1310 +0xe5 fp=0xc000323fe0 sp=0xc000323f50 pc=0x1f1325
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.goexit({})
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/asm_amd64.s:1695 +0x1 fp=0xc000323fe8 sp=0xc000323fe0 pc=0x249481
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): created by runtime.gcBgMarkStartWorkers in goroutine 1
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:1234 +0x1c
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): 
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): goroutine 21 gp=0xc000104a80 m=nil [GC worker (idle)]:
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/proc.go:402 +0xce fp=0xc000325f50 sp=0xc000325f30 pc=0x21260e
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gcBgMarkWorker()
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:1310 +0xe5 fp=0xc000325fe0 sp=0xc000325f50 pc=0x1f1325
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.goexit({})
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/asm_amd64.s:1695 +0x1 fp=0xc000325fe8 sp=0xc000325fe0 pc=0x249481
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): created by runtime.gcBgMarkStartWorkers in goroutine 1
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:1234 +0x1c
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): 
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): goroutine 7 gp=0xc0003e01c0 m=nil [GC worker (idle)]:
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/proc.go:402 +0xce fp=0xc0003e7f50 sp=0xc0003e7f30 pc=0x21260e
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gcBgMarkWorker()
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:1310 +0xe5 fp=0xc0003e7fe0 sp=0xc0003e7f50 pc=0x1f1325
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.goexit({})
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/asm_amd64.s:1695 +0x1 fp=0xc0003e7fe8 sp=0xc0003e7fe0 pc=0x249481
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): created by runtime.gcBgMarkStartWorkers in goroutine 1
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:1234 +0x1c
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): 
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): goroutine 22 gp=0xc000104c40 m=nil [GC worker (idle)]:
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gopark(0x50924d94e0?, 0x0?, 0x0?, 0x0?, 0x0?)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/proc.go:402 +0xce fp=0xc0003e3f50 sp=0xc0003e3f30 pc=0x21260e
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gcBgMarkWorker()
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:1310 +0xe5 fp=0xc0003e3fe0 sp=0xc0003e3f50 pc=0x1f1325
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.goexit({})
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/asm_amd64.s:1695 +0x1 fp=0xc0003e3fe8 sp=0xc0003e3fe0 pc=0x249481
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): created by runtime.gcBgMarkStartWorkers in goroutine 1
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:1234 +0x1c
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): 
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): goroutine 23 gp=0xc000104e00 m=nil [GC worker (idle)]:
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gopark(0x39b6ae0?, 0x1?, 0xc4?, 0x9e?, 0x0?)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/proc.go:402 +0xce fp=0xc0003e5f50 sp=0xc0003e5f30 pc=0x21260e
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gcBgMarkWorker()
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:1310 +0xe5 fp=0xc0003e5fe0 sp=0xc0003e5f50 pc=0x1f1325
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.goexit({})
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/asm_amd64.s:1695 +0x1 fp=0xc0003e5fe8 sp=0xc0003e5fe0 pc=0x249481
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): created by runtime.gcBgMarkStartWorkers in goroutine 1
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:1234 +0x1c
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): 
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): goroutine 8 gp=0xc0003e0540 m=nil [GC worker (idle)]:
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gopark(0x39b6ae0?, 0x1?, 0x70?, 0x66?, 0x0?)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/proc.go:402 +0xce fp=0xc0003e9f50 sp=0xc0003e9f30 pc=0x21260e
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gcBgMarkWorker()
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:1310 +0xe5 fp=0xc0003e9fe0 sp=0xc0003e9f50 pc=0x1f1325
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.goexit({})
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/asm_amd64.s:1695 +0x1 fp=0xc0003e9fe8 sp=0xc0003e9fe0 pc=0x249481
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): created by runtime.gcBgMarkStartWorkers in goroutine 1
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:1234 +0x1c
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): 
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): goroutine 9 gp=0xc0003e0700 m=nil [GC worker (idle)]:
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gopark(0x50924d94e0?, 0x0?, 0x0?, 0x0?, 0x0?)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/proc.go:402 +0xce fp=0xc0003f1f50 sp=0xc0003f1f30 pc=0x21260e
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gcBgMarkWorker()
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:1310 +0xe5 fp=0xc0003f1fe0 sp=0xc0003f1f50 pc=0x1f1325
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.goexit({})
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/asm_amd64.s:1695 +0x1 fp=0xc0003f1fe8 sp=0xc0003f1fe0 pc=0x249481
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): created by runtime.gcBgMarkStartWorkers in goroutine 1
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:1234 +0x1c
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): 
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): goroutine 11 gp=0xc0003e0a80 m=nil [GC worker (idle)]:
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gopark(0x39b6ae0?, 0x3?, 0x0?, 0x0?, 0x0?)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/proc.go:402 +0xce fp=0xc0003edf50 sp=0xc0003edf30 pc=0x21260e
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gcBgMarkWorker()
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:1310 +0xe5 fp=0xc0003edfe0 sp=0xc0003edf50 pc=0x1f1325
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.goexit({})
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/asm_amd64.s:1695 +0x1 fp=0xc0003edfe8 sp=0xc0003edfe0 pc=0x249481
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): created by runtime.gcBgMarkStartWorkers in goroutine 1
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:1234 +0x1c
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): 
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): goroutine 12 gp=0xc0003e0c40 m=nil [GC worker (idle)]:
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gopark(0x39b6ae0?, 0x1?, 0xc4?, 0x9e?, 0x0?)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/proc.go:402 +0xce fp=0xc0003eff50 sp=0xc0003eff30 pc=0x21260e
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gcBgMarkWorker()
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:1310 +0xe5 fp=0xc0003effe0 sp=0xc0003eff50 pc=0x1f1325
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.goexit({})
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/asm_amd64.s:1695 +0x1 fp=0xc0003effe8 sp=0xc0003effe0 pc=0x249481
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): created by runtime.gcBgMarkStartWorkers in goroutine 1
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:1234 +0x1c
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): 
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): goroutine 50 gp=0xc000502000 m=nil [GC worker (idle)]:
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gopark(0x39b6ae0?, 0x1?, 0x0?, 0x0?, 0x0?)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/proc.go:402 +0xce fp=0xc000509f50 sp=0xc000509f30 pc=0x21260e
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gcBgMarkWorker()
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:1310 +0xe5 fp=0xc000509fe0 sp=0xc000509f50 pc=0x1f1325
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.goexit({})
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/asm_amd64.s:1695 +0x1 fp=0xc000509fe8 sp=0xc000509fe0 pc=0x249481
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): created by runtime.gcBgMarkStartWorkers in goroutine 1
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:1234 +0x1c
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): 
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): goroutine 24 gp=0xc000104fc0 m=nil [GC worker (idle)]:
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gopark(0x39b6ae0?, 0x3?, 0xc4?, 0x9e?, 0x0?)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/proc.go:402 +0xce fp=0xc000505f50 sp=0xc000505f30 pc=0x21260e
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gcBgMarkWorker()
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:1310 +0xe5 fp=0xc000505fe0 sp=0xc000505f50 pc=0x1f1325
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.goexit({})
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/asm_amd64.s:1695 +0x1 fp=0xc000505fe8 sp=0xc000505fe0 pc=0x249481
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): created by runtime.gcBgMarkStartWorkers in goroutine 1
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/mgc.go:1234 +0x1c
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): 
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): goroutine 25 gp=0xc000502540 m=nil [chan receive]:
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gopark(0x1db84c0?, 0x0?, 0xa8?, 0x9f?, 0x3168d9?)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/proc.go:402 +0xce fp=0xc000779f18 sp=0xc000779ef8 pc=0x21260e
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.chanrecv(0xc000172900, 0x0, 0x1)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/chan.go:583 +0x3cd fp=0xc000779f90 sp=0xc000779f18 pc=0x1daa0d
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.chanrecv1(0xc0002b6690?, 0x0?)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/chan.go:442 +0x12 fp=0xc000779fb8 sp=0xc000779f90 pc=0x1da612
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): github.com/elastic/elastic-agent/internal/pkg/agent/install.backoffWithContext.func1()
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): github.com/elastic/elastic-agent/internal/pkg/agent/install/uninstall.go:495 +0x2c fp=0xc000779fe0 sp=0xc000779fb8 pc=0xb6d82c
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.goexit({})
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/asm_amd64.s:1695 +0x1 fp=0xc000779fe8 sp=0xc000779fe0 pc=0x249481
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): created by github.com/elastic/elastic-agent/internal/pkg/agent/install.backoffWithContext in goroutine 1
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): github.com/elastic/elastic-agent/internal/pkg/agent/install/uninstall.go:494 +0xa6
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): 
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): goroutine 26 gp=0xc000502700 m=nil [select]:
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.gopark(0xc00077bf98?, 0x3?, 0x0?, 0x0?, 0xc00077bf82?)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/proc.go:402 +0xce fp=0xc00077be28 sp=0xc00077be08 pc=0x21260e
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.selectgo(0xc00077bf98, 0xc00077bf7c, 0x0?, 0x0, 0x0?, 0x1)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/select.go:327 +0x725 fp=0xc00077bf48 sp=0xc00077be28 pc=0x2240e5
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): net/http.setRequestCancel.func4()
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): net/http/client.go:405 +0x85 fp=0xc00077bfe0 sp=0xc00077bf48 pc=0x5a5565
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.goexit({})
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/asm_amd64.s:1695 +0x1 fp=0xc00077bfe8 sp=0xc00077bfe0 pc=0x249481
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): created by net/http.setRequestCancel in goroutine 1
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): net/http/client.go:404 +0x3b5
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): 
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): goroutine 27 gp=0xc0005028c0 m=0 mp=0x392cce0 [syscall (scan), locked to thread]:
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime.cgocall(0x24ac60, 0x392d0e0)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/cgocall.go:157 +0x3e fp=0xc000066860 sp=0xc000066828 pc=0x1d8d5e
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): syscall.SyscallN(0x7ff957180640?, {0xc0000668f8?, 0x3?, 0x0?})
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/syscall_windows.go:544 +0x107 fp=0xc0000668d8 sp=0xc000066860 pc=0x246187
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): syscall.Syscall(0x38f06e0?, 0xc000318380?, 0x9?, 0x1f601e0?, 0x0?)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): runtime/syscall_windows.go:482 +0x35 fp=0xc000066920 sp=0xc0000668d8 pc=0x245dd5
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): syscall.CertCreateCertificateContext(0x10001, 0xc000850163?, 0x840)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): syscall/zsyscall_windows.go:357 +0x65 fp=0xc000066970 sp=0xc000066920 pc=0x2a32e5
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): crypto/x509.createStoreContext(0x11?, 0xc000066d18)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): crypto/x509/root_windows.go:29 +0x65 fp=0xc000066a38 sp=0xc000066970 pc=0x507f85
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): crypto/x509.(*Certificate).systemVerify(0xc000281088, 0xc000066d18)
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): crypto/x509/root_windows.go:204 +0x5e fp=0xc000066bd8 sp=0xc000066a38 pc=0x50901e
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): crypto/x509.(*Certificate).Verify(0xc000281088, {{0x0, 0x0}, 0xc00089e270, 0x0, {0x0, 0x0, 0x0}, {0x0, 0x0, ...}, ...})
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): crypto/x509/verify.go:770 +0x4d4 fp=0xc000066d18 sp=0xc000066bd8 pc=0x50d8d4
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): github.com/elastic/elastic-agent-libs/transport/tlscommon.verifyCertsWithOpts({0x0, 0x0?, 0xc00089e27000?}, {0x0, 0x0?, 0x0}, {{0x0, 0x0}, 0x0, 0x0, ...})
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): crypto/tls/common.go:39 +0xf8 fp=0xc000066d47 sp=0xc000066d18 pc=0x8ecf78
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): created by net/http.(*Transport).queueForDial in goroutine 1
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): net/http/transport.go:1449 +0x3c9
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): install_test.go:476:
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): Error Trace:      C:/Users/windows/agent/testing/integration/install_test.go:476
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): Error:            Received unexpected error:
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): error running uninstall command: exit status 2
>>> (windows-amd64-2022-default) Test output (sudo) (stdout): Test:             TestRepeatedInstallUninstallFleet/TestRepeatedInstallUninstallFleet-6

@cmacknz
Copy link
Member

cmacknz commented Nov 18, 2024

This explains why the standalone uninstall test wasn't finding it.

If we can correlate this to a specific change like the new unenrollment notification, it is acceptable to revert that as a fast work around while we work to get the actual root cause resolved.

@cmacknz
Copy link
Member

cmacknz commented Nov 18, 2024

If we revert #5302 or just disable it by bypassing the block in

elastic-agent/internal/pkg/agent/install/uninstall.go

Lines 147 to 150 in 434986e

if notifyFleet {
notifyFleetAuditUninstall(ctx, log, pt, cfg, ai) //nolint:errcheck // ignore the error as we can't act on it
}
does this still happen?

@cmacknz
Copy link
Member

cmacknz commented Nov 18, 2024

Having an explicit option to skip this step would have helped but it isn't implemented yet.

@blakerouse blakerouse mentioned this issue Nov 18, 2024
Merged
2 tasks
@blakerouse
Copy link
Contributor

I have created a PR that skips this on Windows - #6065

This doesn't actually fix the issue, as the error still happens. Its just that the error happens less likely, 10x less likely.

@cmacknz
Copy link
Member

cmacknz commented Nov 18, 2024 •
edited
Loading

Thanks, if you run 8.15 or 8.14 through the same test does it fail at the same rate? I'm wondering if the revert has put the probability back to where it was before, or if there's still something in 8.16 that makes this more likely.

@blakerouse
Copy link
Contributor

@cmacknz I will run the tests with those versions and report back.

This was referenced Nov 18, 2024
Merged
Merged
@michel-laterman michel-laterman mentioned this issue Nov 19, 2024
Closed
@blakerouse
Copy link
Contributor

I did some more testing on this and I have observed the following:

8.15.0 - Ok
8.15.1 - Ok
8.15.2 - Ok
8.15.3 - Ok
8.16.0 - Fail
8.16.1 - Ok (ran on loop for 24 hours, no failure)
8.16.2-SNAPSHOT [with notifyFleetAuditUninstall] - Fail

It does seem like the notifyFleetAuditUninstall is causing the issue or that it is running longer causing the case more. I am going try a build where I just have uninstall take longer but not have notifyFleetAuditUninstall occurring to see if I can get it to happen.

I original though that notifyFleetAuditUninstall was not the issue and instead it was something else. At the moment everything is pointing to notifyFleetAuditUninstall, but the test to make uninstall to take longer could determine that case.

@blakerouse blakerouse mentioned this issue Dec 9, 2024
Merged
5 tasks
@blakerouse
Copy link
Contributor

What is interesting now is that I cannot reproduce this issue on latest main with notifyFleetAuditUninstall re-enabled on Windows. I have ran it in a loop for 4 hours with no failures. I believe this was a weird issue related to the cross builder.

I have re-enabled it on Windows with the tests I wrote have it up for merge. I will run it a few more times there to see if it shows up there - #6257

@mergify mergify bot mentioned this issue Dec 11, 2024
Merged
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@blakerouse blakerouse

Labels
bug Something isn't working Team:Elastic-Agent Label for the Agent team Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Re-enabled notifyFleetAuditUninstall on Windows blakerouse/elastic-agent
6 participants
@ycombinator @rdner @cmacknz @blakerouse @marc-gr @elasticmachine