Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

upgrading from 0.10 to 0.13 - "concurrent map read and map write" crashes kustomize-controller #341

Closed
caddac opened this issue May 5, 2021 · 1 comment · Fixed by #343
Closed

upgrading from 0.10 to 0.13 - "concurrent map read and map write" crashes kustomize-controller #341

caddac opened this issue May 5, 2021 · 1 comment · Fixed by #343

Comments

@caddac
Copy link

caddac commented May 5, 2021

Describe the bug

During the upgrade from 0.10 to 0.13 I experienced the kustomize-controller crash with a stack trace and the error fatal error: concurrent map read and map write. The controller restarts and successfully reconciles as expected, but only after re-bootstrapping a second time.

This bug does not prevent an upgrade and I am not blocked, but I wanted to report it as there might be an issue in the controller that could pop up later.

To Reproduce

Steps to reproduce the behaviour:

  1. I followed this guide: Upgrade image automation to v1alpha2 APIs flux2#1333
    The issue was intermittent and only occurred on 2 of my 5 clusters.

Expected behavior

The kustomize-controller shouldn't crash due to this error

Additional context

  • Kubernetes version: GKE 1.17.17-gke.3700
  • Git provider: go-git

Below please provide the output of the following commands:

flux --version
  flux version 0.13.2
flux check
► checking prerequisites
✔ kubectl 1.20.2 >=1.18.0-0
✔ Kubernetes 1.17.17-gke.3700 >=1.16.0-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.10.0
✔ image-automation-controller: deployment ready
► ghcr.io/fluxcd/image-automation-controller:v0.9.0
✔ image-reflector-controller: deployment ready
► ghcr.io/fluxcd/image-reflector-controller:v0.9.1
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.12.0
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.13.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.12.1
✔ all checks passed

Here are the kustomize-controller logs captured from the crash pod:

{"level":"info","ts":"2021-05-05T15:44:16.617Z","logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":"2021-05-05T15:44:16.618Z","logger":"setup","msg":"starting manager"}
I0505 15:44:16.618591       7 leaderelection.go:243] attempting to acquire leader lease flux-system/kustomize-controller-leader-election...
{"level":"info","ts":"2021-05-05T15:44:16.619Z","msg":"starting metrics server","path":"/metrics"}
I0505 15:44:16.645125       7 leaderelection.go:253] successfully acquired lease flux-system/kustomize-controller-leader-election
{"level":"info","ts":"2021-05-05T15:44:16.719Z","logger":"controller.kustomization","msg":"Starting EventSource","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","source":"kind source: /, Kind="}
{"level":"info","ts":"2021-05-05T15:44:16.719Z","logger":"controller.kustomization","msg":"Starting EventSource","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","source":"kind source: /, Kind="}
{"level":"info","ts":"2021-05-05T15:44:16.820Z","logger":"controller.kustomization","msg":"Starting EventSource","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","source":"kind source: /, Kind="}
{"level":"info","ts":"2021-05-05T15:44:16.920Z","logger":"controller.kustomization","msg":"Starting Controller","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization"}
{"level":"info","ts":"2021-05-05T15:44:16.920Z","logger":"controller.kustomization","msg":"Starting workers","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","worker count":4}
fatal error: concurrent map read and map write
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/log/log.go:63 +0x45
created by sigs.k8s.io/controller-runtime/pkg/log.init.0
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/log/log.go:62 +0x35
goroutine 54 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0002b2dc0)
    /go/pkg/mod/[email protected]/stats/view/worker.go:154 +0x105
created by go.opencensus.io/stats/view.init.0
    /go/pkg/mod/[email protected]/stats/view/worker.go:32 +0x57
goroutine 275 [select]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc00033cd80)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:742 +0x157
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0003bfc60, 0xc000bd1960)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:71 +0x65
goroutine 59 [IO wait]:
internal/poll.runtime_pollWait(0x7fc452ed4718, 0x72, 0x23cff20)
    /usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000182198, 0x72, 0x23cff00, 0x30bf7a0, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000182180, 0xc000850000, 0x26fa, 0x26fa, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:159 +0x1a5
net.(*netFD).Read(0xc000182180, 0xc000850000, 0x26fa, 0x26fa, 0x203000, 0x6fb87b, 0xc000764f60)
    /usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc0004aa248, 0xc000850000, 0x26fa, 0x26fa, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:182 +0x8e
crypto/tls.(*atLeastReader).Read(0xc001134840, 0xc000850000, 0x26fa, 0x26fa, 0x274, 0x266f, 0xc00001d708)
    /usr/local/go/src/crypto/tls/conn.go:779 +0x62
bytes.(*Buffer).ReadFrom(0xc000765080, 0x23c91e0, 0xc001134840, 0x40b7a5, 0x1e57600, 0x20416c0)
    /usr/local/go/src/bytes/buffer.go:204 +0xb1
crypto/tls.(*Conn).readFromUntil(0xc000764e00, 0x23ccc80, 0xc0004aa248, 0x5, 0xc0004aa248, 0x263)
    /usr/local/go/src/crypto/tls/conn.go:801 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc000764e00, 0x0, 0x0, 0xc0009e84e0)
    /usr/local/go/src/crypto/tls/conn.go:608 +0x115
crypto/tls.(*Conn).readRecord(...)
    /usr/local/go/src/crypto/tls/conn.go:576
crypto/tls.(*Conn).Read(0xc000764e00, 0xc000381000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:1252 +0x15f
bufio.(*Reader).Read(0xc0007fed80, 0xc0007723b8, 0x9, 0x9, 0xc0051765a0, 0xc00001dd18, 0x21b5c00)
    /usr/local/go/src/bufio/bufio.go:227 +0x222
io.ReadAtLeast(0x23c8fa0, 0xc0007fed80, 0xc0007723b8, 0x9, 0x9, 0x9, 0xc0011543f0, 0xc000102050, 0x0)
    /usr/local/go/src/io/io.go:314 +0x87
io.ReadFull(...)
    /usr/local/go/src/io/io.go:333
golang.org/x/net/http2.readFrameHeader(0xc0007723b8, 0x9, 0x9, 0x23c8fa0, 0xc0007fed80, 0x0, 0x0, 0xc000a1b5c0, 0xc00001ddd0)
    /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x89
golang.org/x/net/http2.(*Framer).ReadFrame(0xc000772380, 0xc000a1b5c0, 0x0, 0x0, 0x0)
    /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:492 +0xa5
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc00001dfa8, 0x0, 0x0)
    /go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:1819 +0xd8
golang.org/x/net/http2.(*ClientConn).readLoop(0xc00028a900)
    /go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:1741 +0x6f
created by golang.org/x/net/http2.(*Transport).newClientConn
    /go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:705 +0x6c5
goroutine 292 [chan receive]:
k8s.io/client-go/tools/cache.(*controller).Run.func1(0xc00070a8a0, 0xc00065c2d0)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:130 +0x34
created by k8s.io/client-go/tools/cache.(*controller).Run
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:129 +0xa5
goroutine 271 [chan receive]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start(0xc000acc460, 0x24232c0, 0xc000ae30c0, 0x0, 0x0)
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:228 +0x153
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1(0xc000ae0000, 0x7fc452cfbdf8, 0xc000acc460)
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:681 +0x7e
created by sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:679 +0x73
goroutine 301 [sync.Cond.Wait]:
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:312
sync.runtime_notifyListWait(0xc000b1b910, 0x0)
    /usr/local/go/src/runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc000b1b900)
    /usr/local/go/src/sync/cond.go:56 +0x9d
k8s.io/client-go/util/workqueue.(*Type).Get(0xc000b3aba0, 0x0, 0x0, 0x0)
    /go/pkg/mod/k8s.io/[email protected]/util/workqueue/queue.go:145 +0x89
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000acc460, 0x24232c0, 0xc000ae30c0, 0x0)
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:236 +0x68
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2(0x24232c0, 0xc000ae30c0)
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216 +0x4a
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185 +0x37
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000542f50)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000542f50, 0x23cc720, 0xc0009ec5a0, 0xc000ae3001, 0xc00070a8a0)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000542f50, 0x3b9aca00, 0x0, 0xc00000a901, 0xc00070a8a0)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext(0x24232c0, 0xc000ae30c0, 0xc0003c3a50, 0x3b9aca00, 0x0, 0x100000000000001)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185 +0xa6
k8s.io/apimachinery/pkg/util/wait.UntilWithContext(0x24232c0, 0xc000ae30c0, 0xc0003c3a50, 0x3b9aca00)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:99 +0x57
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:213 +0x45e
goroutine 286 [select]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc000b73400)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:742 +0x157
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc00075a140, 0xc0003c3380)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73 +0x51
    /usr/local/go/src/runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc0004425b0)
    /usr/local/go/src/sync/cond.go:56 +0x9d
golang.org/x/net/http2.(*pipe).Read(0xc0004425a8, 0xc000b41e00, 0x200, 0x200, 0x0, 0x0, 0x0)
    /go/pkg/mod/golang.org/x/[email protected]/http2/pipe.go:65 +0x97
golang.org/x/net/http2.transportResponseBody.Read(0xc000442580, 0xc000b41e00, 0x200, 0x200, 0x0, 0x0, 0x0)
    /go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:2108 +0xaf
encoding/json.(*Decoder).refill(0xc0002b42c0, 0x18, 0x7fc479d02108)
    /usr/local/go/src/encoding/json/stream.go:165 +0xeb
encoding/json.(*Decoder).readValue(0xc0002b42c0, 0x0, 0x0, 0x1e04d60)
    /usr/local/go/src/encoding/json/stream.go:140 +0x1ff
encoding/json.(*Decoder).Decode(0xc0002b42c0, 0x1e3c540, 0xc00022ec60, 0xc00006f000, 0x0)
    /usr/local/go/src/encoding/json/stream.go:63 +0x79
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc000a1bd70, 0xc000480800, 0x400, 0x400, 0xc0003c36c0, 0x7fc4530a1430, 0x38)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/framer/framer.go:152 +0x1a8
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc000a2b720, 0x0, 0x23df740, 0xc000356c40, 0x0, 0xa, 0x2, 0x100000000000000, 0x0)
    /go/pkg/mod/k8s.io/[email protected]/pkg/runtime/serializer/streaming/streaming.go:77 +0x89
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc00022ec20, 0xc00039a790, 0xc0001caeb8, 0xc00039a780, 0x12e35cb, 0xc0004cff40, 0xc000413160)
    /go/pkg/mod/k8s.io/[email protected]/rest/watch/decoder.go:49 +0x6e
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc000356b40)
    /go/pkg/mod/k8s.io/[email protected]/pkg/watch/streamwatcher.go:104 +0x14a
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
    /go/pkg/mod/k8s.io/[email protected]/pkg/watch/streamwatcher.go:71 +0xbe
goroutine 254 [sync.Cond.Wait]:
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:312
sync.runtime_notifyListWait(0xc000340720, 0x6)
    /usr/local/go/src/runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc000340710)
    /usr/local/go/src/sync/cond.go:56 +0x9d
golang.org/x/net/http2.(*pipe).Read(0xc000340708, 0xc0007cb001, 0xdff, 0xdff, 0x0, 0x0, 0x0)
    /go/pkg/mod/golang.org/x/[email protected]/http2/pipe.go:65 +0x97
golang.org/x/net/http2.transportResponseBody.Read(0xc0003406e0, 0xc0007cb001, 0xdff, 0xdff, 0x0, 0x0, 0x0)
    /go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:2108 +0xaf
encoding/json.(*Decoder).refill(0xc0000cb8c0, 0xa, 0x9)
    /usr/local/go/src/encoding/json/stream.go:165 +0xeb
encoding/json.(*Decoder).readValue(0xc0000cb8c0, 0x0, 0x0, 0x1e04d60)
    /usr/local/go/src/encoding/json/stream.go:140 +0x1ff
encoding/json.(*Decoder).Decode(0xc0000cb8c0, 0x1e3c540, 0xc00087c940, 0x0, 0x0)
    /usr/local/go/src/encoding/json/stream.go:63 +0x79
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc00048d8f0, 0xc000906000, 0x1000, 0x1300, 0xc0007f5d98, 0x7fc452e08868, 0x38)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/framer/framer.go:152 +0x1a8
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc00049c4b0, 0x0, 0x23df740, 0xc000933140, 0x0, 0x0, 0x464b00, 0xc00070ac00, 0xc0007f5e80)
    /go/pkg/mod/k8s.io/[email protected]/pkg/runtime/serializer/streaming/streaming.go:77 +0x89
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc000527ea0, 0xc0007f5f18, 0x8, 0x23d9200, 0xc000ae1340, 0x0, 0x0)
    /go/pkg/mod/k8s.io/[email protected]/rest/watch/decoder.go:49 +0x6e
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc0009aa940)
    /go/pkg/mod/k8s.io/[email protected]/pkg/watch/streamwatcher.go:104 +0x14a
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
    /go/pkg/mod/k8s.io/[email protected]/pkg/watch/streamwatcher.go:71 +0xbe
goroutine 302 [sync.Cond.Wait]:
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:312
sync.runtime_notifyListWait(0xc000b1b910, 0x1)
    /usr/local/go/src/runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc000b1b900)
    /usr/local/go/src/sync/cond.go:56 +0x9d
k8s.io/client-go/util/workqueue.(*Type).Get(0xc000b3aba0, 0x0, 0x0, 0xc000702300)
    /go/pkg/mod/k8s.io/[email protected]/util/workqueue/queue.go:145 +0x89
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000acc460, 0x24232c0, 0xc000ae30c0, 0x0)
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:236 +0x68
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2(0x24232c0, 0xc000ae30c0)
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216 +0x4a
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185 +0x37
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00024ef50)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00024ef50, 0x23cc720, 0xc0009ec5d0, 0xc000ae3001, 0xc00070a8a0)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00024ef50, 0x3b9aca00, 0x0, 0x3b9aca01, 0xc00070a8a0)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext(0x24232c0, 0xc000ae30c0, 0xc0003c3a60, 0x3b9aca00, 0x0, 0x10000c000981901)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185 +0xa6
k8s.io/apimachinery/pkg/util/wait.UntilWithContext(0x24232c0, 0xc000ae30c0, 0xc0003c3a60, 0x3b9aca00)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:99 +0x57
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:213 +0x45e
goroutine 358 [select]:
net/http.(*persistConn).writeLoop(0xc00091e5a0)
    /usr/local/go/src/net/http/transport.go:2346 +0x11c
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1716 +0xcdc
goroutine 274 [chan receive]:
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:772 +0x5d
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00024ef60)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00083ef60, 0x23cc720, 0xc0008f9320, 0x1de8c01, 0xc0006c4900)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00024ef60, 0x3b9aca00, 0x0, 0x7fc452d36201, 0xc0006c4900)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run(0xc00033cd80)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:771 +0x95
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0003bfc60, 0xc000bd1950)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:71 +0x65
goroutine 303 [runnable]:
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:56 +0x2e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0009c2150, 0xc00099b6e0)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:71 +0x65
goroutine 235 [chan receive]:
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).serveHealthProbes(0xc000ae0000)
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:431 +0xea
created by sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).Start
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:478 +0x44a
goroutine 240 [select]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func2(0xc0009c81c0, 0xc00070a8a0, 0xc0006c5bc0, 0xc000b3a240)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:373 +0x16f
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:367 +0x2c5
goroutine 260 [chan receive]:
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1(0x23dfb80, 0xc00036d380, 0xc000bd1130)
    /go/pkg/mod/k8s.io/[email protected]/tools/record/event.go:301 +0xaa
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
    /go/pkg/mod/k8s.io/[email protected]/tools/record/event.go:299 +0x6e
goroutine 291 [chan receive]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc00075a7e0, 0xc0006c50e0)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:628 +0x53
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:56 +0x2e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0004d0af0, 0xc00022e6a0)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:71 +0x65
goroutine 198 [chan receive]:
sigs.k8s.io/controller-runtime/pkg/cache/internal.(*InformersMap).Start(0xc00094bba0, 0x24232c0, 0xc000ae30c0, 0x0, 0x0)
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/cache/internal/deleg_map.go:68 +0xf9
sigs.k8s.io/controller-runtime/pkg/cluster.(*cluster).Start(0xc000946e00, 0x24232c0, 0xc000ae30c0, 0x0, 0x0)
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/cluster/internal.go:127 +0x8a
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1(0xc000ae0000, 0x7fc452cfc640, 0xc000946e00)
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:681 +0x7e
created by sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:679 +0x73
goroutine 233 [chan receive]:
sigs.k8s.io/controller-runtime/pkg/manager/signals.SetupSignalHandler.func1(0xc000a959e0, 0xc000a675f0)
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/signals/signal.go:38 +0x34
created by sigs.k8s.io/controller-runtime/pkg/manager/signals.SetupSignalHandler
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/signals/signal.go:37 +0xe8
goroutine 293 [select]:
k8s.io/client-go/tools/cache.(*Reflector).watchHandler(0xc000772460, 0xc01ccd343123e525, 0x276ecbc9, 0x327c4c0, 0x23dfb40, 0xc000356b40, 0xc00001bbc8, 0xc000980ea0, 0xc00070a8a0, 0x0, ...)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:463 +0x1a5
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc000772460, 0xc00070a8a0, 0x0, 0x0)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:427 +0x6ac
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:221 +0x38
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000542ef0)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00001bef0, 0x23cc700, 0xc000a2b590, 0x1, 0xc00070a8a0)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xad
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc000772460, 0xc00070a8a0)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:220 +0x1c5
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:56 +0x2e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0004d0b10, 0xc00022e6e0)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:71 +0x65
goroutine 298 [chan receive]:
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:772 +0x5d
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000543760)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000543760, 0x23cc720, 0xc0009ec600, 0x1de8c01, 0xc0006c5ec0)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000543760, 0x3b9aca00, 0x0, 0x1, 0xc0006c5ec0)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run(0xc000b73500)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:771 +0x95
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc00075a840, 0xc0003c39c0)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:71 +0x65
goroutine 256 [select]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func2(0xc000772460, 0xc00070a8a0, 0xc00070aea0, 0xc000980ea0)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:373 +0x16f
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:367 +0x2c5
goroutine 360 [select]:
net/http.(*persistConn).writeLoop(0xc00070f200)
    /usr/local/go/src/net/http/transport.go:2346 +0x11c
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1716 +0xcdc
goroutine 272 [chan receive]:
k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc000b3aba0)
    /go/pkg/mod/k8s.io/[email protected]/util/workqueue/queue.go:198 +0xac
created by k8s.io/client-go/util/workqueue.newQueue
    /go/pkg/mod/k8s.io/[email protected]/util/workqueue/queue.go:58 +0x135
goroutine 284 [select]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func2(0xc0007720e0, 0xc00070a8a0, 0xc0006c4d80, 0xc000b3b800)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:373 +0x16f
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:367 +0x2c5
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*controller).Run(0xc000abfd40, 0xc00070a8a0)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:154 +0x2e5
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc000acc960, 0xc00070a8a0)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:410 +0x42a
created by sigs.k8s.io/controller-runtime/pkg/cache/internal.(*specificInformersMap).addInformerToMap
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/cache/internal/informers_map.go:229 +0x5a5
goroutine 208 [select]:
k8s.io/client-go/tools/cache.(*Reflector).watchHandler(0xc0009c81c0, 0xc01ccd34259f2fb8, 0x1bea165c, 0x327c4c0, 0x23dfb40, 0xc0009ab180, 0xc000fddbc8, 0xc000b3a240, 0xc00070a8a0, 0x0, ...)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:463 +0x1a5
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0009c81c0, 0xc00070a8a0, 0x0, 0x0)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:427 +0x6ac
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:221 +0x38
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0004b76f0)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000fddef0, 0x23cc700, 0xc00049cfa0, 0x1, 0xc00070a8a0)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xad
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0009c81c0, 0xc00070a8a0)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:220 +0x1c5
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:56 +0x2e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0009c2160, 0xc00099b720)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:71 +0x65
goroutine 234 [chan receive]:
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).serveMetrics(0xc000ae0000)
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:393 +0x1fa
created by sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).Start
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:473 +0x474
goroutine 287 [sync.Cond.Wait]:
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:312
sync.runtime_notifyListWait(0xc000accc08, 0x0)
    /usr/local/go/src/runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc000accbf8)
    /usr/local/go/src/sync/cond.go:56 +0x9d
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc000accbe0, 0xc0003c3580, 0x0, 0x0, 0x0, 0x0)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/delta_fifo.go:493 +0x98
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc00065c2d0)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:183 +0x42
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0009cee68)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0009cee68, 0x23cc720, 0xc000a1b980, 0xc0004d0b01, 0xc00070a8a0)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0009cee68, 0x3b9aca00, 0x0, 0xc00000a901, 0xc00070a8a0)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*controller).Run(0xc00065c2d0, 0xc00070a8a0)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:154 +0x2e5
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc000accb40, 0xc00070a8a0)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:410 +0x42a
created by sigs.k8s.io/controller-runtime/pkg/cache/internal.(*specificInformersMap).addInformerToMap
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/cache/internal/informers_map.go:229 +0x5a5
goroutine 246 [sync.Cond.Wait]:
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:312
sync.runtime_notifyListWait(0xc000ae54e0, 0x2)
    /usr/local/go/src/runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc000ae54d0)
    /usr/local/go/src/sync/cond.go:56 +0x9d
golang.org/x/net/http2.(*pipe).Read(0xc000ae54c8, 0xc00017a001, 0xdff, 0xdff, 0x0, 0x0, 0x0)
    /go/pkg/mod/golang.org/x/[email protected]/http2/pipe.go:65 +0x97
golang.org/x/net/http2.transportResponseBody.Read(0xc000ae54a0, 0xc00017a001, 0xdff, 0xdff, 0x0, 0x0, 0x0)
    /go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:2108 +0xaf
encoding/json.(*Decoder).refill(0xc0009fe420, 0xa, 0x9)
    /usr/local/go/src/encoding/json/stream.go:165 +0xeb
encoding/json.(*Decoder).readValue(0xc0009fe420, 0x0, 0x0, 0x1e04d60)
    /usr/local/go/src/encoding/json/stream.go:140 +0x1ff
encoding/json.(*Decoder).Decode(0xc0009fe420, 0x1e3c540, 0xc00094adc0, 0x0, 0x0)
    /usr/local/go/src/encoding/json/stream.go:63 +0x79
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc0009ed8c0, 0xc00032b300, 0x1000, 0x1300, 0xc000395d98, 0x7fc452e09b88, 0x38)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/framer/framer.go:152 +0x1a8
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc00049d310, 0x0, 0x23df740, 0xc0006f8100, 0x0, 0x0, 0x464b00, 0xc00070ad20, 0xc000395e80)
    /go/pkg/mod/k8s.io/[email protected]/pkg/runtime/serializer/streaming/streaming.go:77 +0x89
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc0009e9a60, 0xc000395f18, 0x8, 0x23d9040, 0xc000343600, 0x0, 0x0)
    /go/pkg/mod/k8s.io/[email protected]/rest/watch/decoder.go:49 +0x6e
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc0009ab180)
    /go/pkg/mod/k8s.io/[email protected]/pkg/watch/streamwatcher.go:104 +0x14a
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
    /go/pkg/mod/k8s.io/[email protected]/pkg/watch/streamwatcher.go:71 +0xbe
goroutine 299 [select]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc000b73500)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:742 +0x157
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc00075a840, 0xc0003c39d0)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:71 +0x65
goroutine 259 [chan receive]:
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1(0x23dfb80, 0xc00036d350, 0xc00036d320)
    /go/pkg/mod/k8s.io/[email protected]/tools/record/event.go:301 +0xaa
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
    /go/pkg/mod/k8s.io/[email protected]/tools/record/event.go:299 +0x6e
goroutine 207 [chan receive]:
k8s.io/client-go/tools/cache.(*controller).Run.func1(0xc00070a8a0, 0xc000983d40)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:130 +0x34
created by k8s.io/client-go/tools/cache.(*controller).Run
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:129 +0xa5
stream closed```
@stefanprodan stefanprodan transferred this issue from fluxcd/flux2 May 6, 2021
@stefanprodan
Copy link
Member

stefanprodan commented May 6, 2021 •
edited
Loading

Duplicate of: #310
Upstream bug: kubernetes-sigs/kustomize#3659

@stefanprodan stefanprodan mentioned this issue May 11, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Upgrade controller to Kustomize v4 fluxcd/kustomize-controller
2 participants
@stefanprodan @caddac