Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"Fatal error: concurrent map writes" using KOCACHE #579

Closed
jdolitsky opened this issue Feb 2, 2022 · 5 comments · Fixed by #580
Closed

"Fatal error: concurrent map writes" using KOCACHE #579

jdolitsky opened this issue Feb 2, 2022 · 5 comments · Fixed by #580

Comments

@jdolitsky
Copy link

jdolitsky commented Feb 2, 2022 •
edited
Loading

When using KOCACHE, I intermittently get a panic error "Fatal error: concurrent map writes"

The source appears to be https://github.com/google/ko/blob/e961890072fbfda7b8172cd0614a107f76716c9a/pkg/build/cache.go#L50

Here is the full stack trace:

fatal error: concurrent map writes

goroutine 105 [running]:
runtime.throw({0x1034fb00a, 0x15})
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/runtime/panic.go:1198 +0x54 fp=0x140004390c0 sp=0x14000439090 pc=0x102de2804
runtime.mapassign_faststr(0x1036abe40, 0x1400025ddd0, {0x1400010ecc0, 0x5d})
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/runtime/map_faststr.go:211 +0x3e8 fp=0x14000439130 sp=0x140004390c0 pc=0x102dbfdc8
github.com/google/ko/pkg/build.(*layerCache).readBuildToDiff(0x140001ae3a8, {0x1400010ecc0, 0x5d})
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/build/cache.go:195 +0x1c8 fp=0x140004391c0 sp=0x14000439130 pc=0x103287fa8
github.com/google/ko/pkg/build.(*layerCache).getMeta(0x140001ae3a8, {0x1037acb20, 0x140001d18c0}, {0x1400010ecc0, 0x5d})
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/build/cache.go:79 +0x88 fp=0x140004392f0 sp=0x140004391c0 pc=0x103286f18
github.com/google/ko/pkg/build.(*layerCache).get(0x140001ae3a8, {0x1037acb20, 0x140001d18c0}, {0x1400010ecc0, 0x5d}, 0x1400040c540)
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/build/cache.go:50 +0x68 fp=0x140004393a0 sp=0x140004392f0 pc=0x103286c18
github.com/google/ko/pkg/build.(*gobuild).buildOne(0x14000222a00, {0x1037acb20, 0x140001d18c0}, {0x140006180c0, 0x33}, {0x1037bf9d8, 0x140002821c0}, 0x1400020e1c0)
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/build/gobuild.go:725 +0x7a0 fp=0x140004399a0 sp=0x140004393a0 pc=0x10328d960
github.com/google/ko/pkg/build.(*gobuild).buildAll(0x14000222a00, {0x1037acb20, 0x140001d18c0}, {0x140006180c0, 0x33}, {0x1037b7e10, 0x140003b8300})
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/build/gobuild.go:922 +0x2d4 fp=0x14000439c60 sp=0x140004399a0 pc=0x10328fa84
github.com/google/ko/pkg/build.(*gobuild).Build(0x14000222a00, {0x1037acb20, 0x140001d18c0}, {0x140006180c0, 0x33})
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/build/gobuild.go:879 +0x628 fp=0x14000439dc0 sp=0x14000439c60 pc=0x10328f538
github.com/google/ko/pkg/build.(*gobuilds).Build(0x1400025de30, {0x1037acb20, 0x140001d18c0}, {0x140006180c0, 0x33})
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/build/gobuilds.go:91 +0x90 fp=0x14000439f00 sp=0x14000439dc0 pc=0x103291160
github.com/google/ko/pkg/build.(*Caching).Build.func1.1()
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/build/shared.go:60 +0x54 fp=0x14000439f60 sp=0x14000439f00 pc=0x1032924d4
github.com/google/ko/pkg/build.newFuture.func1(0x14000612a80, 0x140000a2180)
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/build/future.go:27 +0x28 fp=0x14000439fc0 sp=0x14000439f60 pc=0x103288798
runtime.goexit()
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/runtime/asm_arm64.s:1133 +0x4 fp=0x14000439fc0 sp=0x14000439fc0 pc=0x102e15664
created by github.com/google/ko/pkg/build.newFuture
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/build/future.go:26 +0x5c

goroutine 1 [select]:
github.com/google/ko/pkg/commands.resolveFilesToWriter({0x1037ad3e0, 0x14000516c40}, 0x140001ccce0, {0x1037a3000, 0x140001ccf00}, 0x14000403000, 0x1400045b6b0, {0x1037a4658, 0x140001b2008})
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/commands/resolver.go:349 +0x4b0
github.com/google/ko/pkg/commands.addResolve.func1(0x14000518500, {0x14000181e00, 0x0, 0x5})
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/commands/resolve.go:70 +0x1d8
github.com/spf13/cobra.(*Command).execute(0x14000518500, {0x14000181db0, 0x5, 0x5})
	/Users/jdolitsky/go/pkg/mod/github.com/spf13/[email protected]/command.go:856 +0x668
github.com/spf13/cobra.(*Command).ExecuteC(0x14000509680)
	/Users/jdolitsky/go/pkg/mod/github.com/spf13/[email protected]/command.go:974 +0x410
github.com/spf13/cobra.(*Command).Execute(...)
	/Users/jdolitsky/go/pkg/mod/github.com/spf13/[email protected]/command.go:902
github.com/spf13/cobra.(*Command).ExecuteContext(...)
	/Users/jdolitsky/go/pkg/mod/github.com/spf13/[email protected]/command.go:895
main.main()
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/main.go:35 +0x160

goroutine 20 [syscall]:
os/signal.signal_recv()
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/runtime/sigqueue.go:166 +0x34
os/signal.loop()
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/os/signal/signal_unix.go:24 +0x20
created by os/signal.Notify.func1.1
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/os/signal/signal.go:151 +0x38

goroutine 5 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x103cc47a0)
	/Users/jdolitsky/go/pkg/mod/k8s.io/klog/[email protected]/klog.go:1181 +0x70
created by k8s.io/klog/v2.init.0
	/Users/jdolitsky/go/pkg/mod/k8s.io/klog/[email protected]/klog.go:420 +0x150

goroutine 21 [select]:
os/signal.NotifyContext.func1(0x14000516c40)
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/os/signal/signal.go:288 +0x7c
created by os/signal.NotifyContext
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/os/signal/signal.go:287 +0x180

goroutine 34 [semacquire]:
sync.runtime_Semacquire(0x14000272130)
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/runtime/sema.go:56 +0x38
sync.(*WaitGroup).Wait(0x14000272128)
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/sync/waitgroup.go:130 +0xa4
golang.org/x/sync/errgroup.(*Group).Wait(0x14000272120)
	/Users/jdolitsky/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:40 +0x30
github.com/google/ko/pkg/resolve.ImageReferences({0x1037acb20, 0x140001d18c0}, {0x14000130090, 0x1, 0x1}, {0x1037a5410, 0x140005151a0}, {0x1037a3000, 0x140001ccf00})
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/resolve/resolve.go:70 +0x274
github.com/google/ko/pkg/commands.resolveFile({0x1037acb20, 0x140001d18c0}, {0x14000124000, 0x1f}, {0x1037a5410, 0x140005151a0}, {0x1037a3000, 0x140001ccf00}, 0x1400045b6b0)
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/commands/resolver.go:482 +0x580
github.com/google/ko/pkg/commands.resolveFilesToWriter.func2()
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/commands/resolver.go:374 +0x154
golang.org/x/sync/errgroup.(*Group).Go.func1(0x140002728d0, 0x1400012c000)
	/Users/jdolitsky/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57 +0x58
created by golang.org/x/sync/errgroup.(*Group).Go
	/Users/jdolitsky/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:54 +0x60

goroutine 35 [semacquire]:
sync.runtime_Semacquire(0x14000612a60)
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/runtime/sema.go:56 +0x38
sync.(*WaitGroup).Wait(0x14000612a58)
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/sync/waitgroup.go:130 +0xa4
golang.org/x/sync/errgroup.(*Group).Wait(0x14000612a50)
	/Users/jdolitsky/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:40 +0x30
github.com/google/ko/pkg/resolve.ImageReferences({0x1037acb20, 0x140001d18c0}, {0x1400061e078, 0x1, 0x1}, {0x1037a5410, 0x1400012e000}, {0x1037a3000, 0x140001ccf00})
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/resolve/resolve.go:70 +0x274
github.com/google/ko/pkg/commands.resolveFile({0x1037acb20, 0x140001d18c0}, {0x14000128000, 0x21}, {0x1037a5410, 0x1400012e000}, {0x1037a3000, 0x140001ccf00}, 0x1400045b6b0)
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/commands/resolver.go:482 +0x580
github.com/google/ko/pkg/commands.resolveFilesToWriter.func2()
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/commands/resolver.go:374 +0x154
golang.org/x/sync/errgroup.(*Group).Go.func1(0x140002728d0, 0x1400012c070)
	/Users/jdolitsky/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57 +0x58
created by golang.org/x/sync/errgroup.(*Group).Go
	/Users/jdolitsky/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:54 +0x60

goroutine 147 [IO wait]:
internal/poll.runtime_pollWait(0x12aed6be0, 0x72)
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/runtime/netpoll.go:234 +0xa4
internal/poll.(*pollDesc).wait(0x14000210398, 0x72, 0x0)
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/internal/poll/fd_poll_runtime.go:84 +0x38
internal/poll.(*pollDesc).waitRead(...)
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x14000210380, {0x140004e7000, 0x1670, 0x1670})
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/internal/poll/fd_unix.go:167 +0x1dc
net.(*netFD).Read(0x14000210380, {0x140004e7000, 0x1670, 0x1670})
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/net/fd_posix.go:56 +0x44
net.(*conn).Read(0x14000010128, {0x140004e7000, 0x1670, 0x1670})
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/net/net.go:183 +0x4c
crypto/tls.(*atLeastReader).Read(0x14000292618, {0x140004e7000, 0x1670, 0x1670})
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/crypto/tls/conn.go:777 +0x58
bytes.(*Buffer).ReadFrom(0x140001a6278, {0x1037964c0, 0x14000292618})
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/bytes/buffer.go:204 +0xa4
crypto/tls.(*Conn).readFromUntil(0x140001a6000, {0x12af8c298, 0x14000010128}, 0x5)
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/crypto/tls/conn.go:799 +0xe0
crypto/tls.(*Conn).readRecordOrCCS(0x140001a6000, 0x0)
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/crypto/tls/conn.go:606 +0xf4
crypto/tls.(*Conn).readRecord(...)
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/crypto/tls/conn.go:574
crypto/tls.(*Conn).Read(0x140001a6000, {0x1400047f000, 0x1000, 0x1000})
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/crypto/tls/conn.go:1277 +0x164
bufio.(*Reader).Read(0x14000347a40, {0x140001de3b8, 0x9, 0x9})
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/bufio/bufio.go:227 +0x20c
io.ReadAtLeast({0x1037962e0, 0x14000347a40}, {0x140001de3b8, 0x9, 0x9}, 0x9)
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/io/io.go:328 +0xa0
io.ReadFull(...)
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/io/io.go:347
net/http.http2readFrameHeader({0x140001de3b8, 0x9, 0x9}, {0x1037962e0, 0x14000347a40})
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/net/http/h2_bundle.go:1555 +0x5c
net/http.(*http2Framer).ReadFrame(0x140001de380)
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/net/http/h2_bundle.go:1813 +0x90
net/http.(*http2clientConnReadLoop).run(0x140005bafb0)
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/net/http/h2_bundle.go:8608 +0x10c
net/http.(*http2ClientConn).readLoop(0x140002f2180)
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/net/http/h2_bundle.go:8531 +0x58
created by net/http.(*http2Transport).newClientConn
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/net/http/h2_bundle.go:7325 +0xb7c

goroutine 104 [chan receive]:
github.com/google/ko/pkg/build.(*future).Get(0x14000612ab0)
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/build/future.go:55 +0x4c
github.com/google/ko/pkg/build.(*Caching).Build(0x140001ccce0, {0x1037acb20, 0x140001d18c0}, {0x140006180c0, 0x33})
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/build/shared.go:66 +0x54
github.com/google/ko/pkg/build.(*Recorder).Build(0x1400012e000, {0x1037acb20, 0x140001d18c0}, {0x140006180c0, 0x33})
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/build/recorder.go:51 +0x70
github.com/google/ko/pkg/resolve.ImageReferences.func1()
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/resolve/resolve.go:58 +0x78
golang.org/x/sync/errgroup.(*Group).Go.func1(0x14000612a50, 0x140001801e0)
	/Users/jdolitsky/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57 +0x58
created by golang.org/x/sync/errgroup.(*Group).Go
	/Users/jdolitsky/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:54 +0x60

goroutine 52 [chan receive]:
github.com/google/ko/pkg/build.(*future).Get(0x14000272180)
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/build/future.go:55 +0x4c
github.com/google/ko/pkg/build.(*Caching).Build(0x140001ccce0, {0x1037acb20, 0x140001d18c0}, {0x14000154180, 0x31})
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/build/shared.go:66 +0x54
github.com/google/ko/pkg/build.(*Recorder).Build(0x140005151a0, {0x1037acb20, 0x140001d18c0}, {0x14000154180, 0x31})
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/build/recorder.go:51 +0x70
github.com/google/ko/pkg/resolve.ImageReferences.func1()
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/resolve/resolve.go:58 +0x78
golang.org/x/sync/errgroup.(*Group).Go.func1(0x14000272120, 0x14000296050)
	/Users/jdolitsky/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57 +0x58
created by golang.org/x/sync/errgroup.(*Group).Go
	/Users/jdolitsky/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:54 +0x60

goroutine 53 [runnable]:
syscall.syscall(0x102e29e00, 0x140002e41b0, 0x140006d4038, 0x0)
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/runtime/sys_darwin.go:22 +0x1c
syscall.Stat({0x140002e4198, 0x14}, 0x140006d4038)
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/syscall/zsyscall_darwin_arm64.go:1924 +0xec
os.statNolog.func1(...)
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/os/stat_unix.go:33
os.ignoringEINTR(...)
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/os/file_posix.go:246
os.statNolog({0x140002e4198, 0x14})
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/os/stat_unix.go:32 +0x60
os.Stat({0x140002e4198, 0x14})
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/os/stat.go:13 +0x44
os/exec.findExecutable({0x140002e4198, 0x14})
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/os/exec/lp_unix.go:22 +0x30
os/exec.LookPath({0x1034ef6a6, 0x2})
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/os/exec/lp_unix.go:55 +0x2d4
os/exec.Command({0x1034ef6a6, 0x2}, {0x1400016b2b8, 0x3, 0x3})
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/os/exec/exec.go:175 +0x1e0
os/exec.CommandContext(...)
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/os/exec/exec.go:193
github.com/google/ko/pkg/build.goversionm({0x1037acb20, 0x140001d18c0}, {0x14000182ba0, 0x5b}, {0x140001ec180, 0x12}, {0x1037bf888, 0x140001b80e0})
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/build/gobuild.go:296 +0xb0
github.com/google/ko/pkg/build.spdx.func1({0x1037acb20, 0x140001d18c0}, {0x14000182ba0, 0x5b}, {0x140001ec180, 0x12}, {0x1037bf888, 0x140001b80e0})
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/build/gobuild.go:310 +0x70
github.com/google/ko/pkg/build.(*gobuild).buildOne(0x14000222a00, {0x1037acb20, 0x140001d18c0}, {0x14000154180, 0x31}, {0x1037bf9d8, 0x140001cc540}, 0x140001b8150)
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/build/gobuild.go:785 +0x11a0
github.com/google/ko/pkg/build.(*gobuild).buildAll(0x14000222a00, {0x1037acb20, 0x140001d18c0}, {0x14000154180, 0x31}, {0x1037b7e10, 0x140006ae120})
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/build/gobuild.go:922 +0x2d4
github.com/google/ko/pkg/build.(*gobuild).Build(0x14000222a00, {0x1037acb20, 0x140001d18c0}, {0x14000154180, 0x31})
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/build/gobuild.go:879 +0x628
github.com/google/ko/pkg/build.(*gobuilds).Build(0x1400025de30, {0x1037acb20, 0x140001d18c0}, {0x14000154180, 0x31})
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/build/gobuilds.go:91 +0x90
github.com/google/ko/pkg/build.(*Caching).Build.func1.1()
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/build/shared.go:60 +0x54
github.com/google/ko/pkg/build.newFuture.func1(0x14000272150, 0x14000182180)
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/build/future.go:27 +0x28
created by github.com/google/ko/pkg/build.newFuture
	/Users/jdolitsky/go/pkg/mod/github.com/google/[email protected]/pkg/build/future.go:26 +0x5c

goroutine 113 [IO wait]:
internal/poll.runtime_pollWait(0x12aed6db0, 0x72)
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/runtime/netpoll.go:234 +0xa4
internal/poll.(*pollDesc).wait(0x14000160198, 0x72, 0x0)
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/internal/poll/fd_poll_runtime.go:84 +0x38
internal/poll.(*pollDesc).waitRead(...)
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x14000160180, {0x140001e5500, 0x14c2, 0x14c2})
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/internal/poll/fd_unix.go:167 +0x1dc
net.(*netFD).Read(0x14000160180, {0x140001e5500, 0x14c2, 0x14c2})
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/net/fd_posix.go:56 +0x44
net.(*conn).Read(0x1400061e268, {0x140001e5500, 0x14c2, 0x14c2})
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/net/net.go:183 +0x4c
crypto/tls.(*atLeastReader).Read(0x14000408390, {0x140001e5500, 0x14c2, 0x14c2})
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/crypto/tls/conn.go:777 +0x58
bytes.(*Buffer).ReadFrom(0x140000b8cf8, {0x1037964c0, 0x14000408390})
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/bytes/buffer.go:204 +0xa4
crypto/tls.(*Conn).readFromUntil(0x140000b8a80, {0x12af8c298, 0x1400061e268}, 0x5)
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/crypto/tls/conn.go:799 +0xe0
crypto/tls.(*Conn).readRecordOrCCS(0x140000b8a80, 0x0)
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/crypto/tls/conn.go:606 +0xf4
crypto/tls.(*Conn).readRecord(...)
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/crypto/tls/conn.go:574
crypto/tls.(*Conn).Read(0x140000b8a80, {0x140004fb000, 0x1000, 0x1000})
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/crypto/tls/conn.go:1277 +0x164
bufio.(*Reader).Read(0x140003474a0, {0x1400032d8b8, 0x9, 0x9})
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/bufio/bufio.go:227 +0x20c
io.ReadAtLeast({0x1037962e0, 0x140003474a0}, {0x1400032d8b8, 0x9, 0x9}, 0x9)
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/io/io.go:328 +0xa0
io.ReadFull(...)
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/io/io.go:347
net/http.http2readFrameHeader({0x1400032d8b8, 0x9, 0x9}, {0x1037962e0, 0x140003474a0})
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/net/http/h2_bundle.go:1555 +0x5c
net/http.(*http2Framer).ReadFrame(0x1400032d880)
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/net/http/h2_bundle.go:1813 +0x90
net/http.(*http2clientConnReadLoop).run(0x1400054dfb0)
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/net/http/h2_bundle.go:8608 +0x10c
net/http.(*http2ClientConn).readLoop(0x14000220480)
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/net/http/h2_bundle.go:8531 +0x58
created by net/http.(*http2Transport).newClientConn
	/opt/homebrew/Cellar/go/1.17.6/libexec/src/net/http/h2_bundle.go:7325 +0xb7c
Error: exit status 2
@imjasonh
Copy link
Member

imjasonh commented Feb 2, 2022

Cool! I've never seen this before. Were you doing ko apply or resolve that pushes and caches a bunch of images? Multi-platform images?

I think we should either add a mutex or maybe use a sync.Map.

@jdolitsky
Copy link
Author

@imjasonh - it's using ko resolve using the new --image-refs flag, pointing to a text file with 16 images :)

@jdolitsky
Copy link
Author

Also worth noting is --platform=linux/arm64

@imjasonh
Copy link
Member

imjasonh commented Feb 2, 2022

I'm able to reproduce with

ko resolve -f config/ --platform=all --image-refs=out.txt > resolved.yaml

on Tekton's config YAML, which builds "only" about 10 images.

The --platform shouldn't really matter for this, except that that map gets written to for every image that gets built, so 16 single-platform images is actually fewer potential concurrent map writes than Tekton's 10 images each building ~5 images each.

--image-refs also shouldn't matter, because that's only reporting the output images built by ko, and not telling it what images to build.

If I lock and defer-unlock the sync.Mutex in pkg/build/cache.go, right before c.readBuildToDiff, I don't see the panic anymore. I'll test some more and send a PR.

btw, building Tekton's 10 images for ~5 platforms took ~9 minutes without KOCACHE, which is already incredible, and ~3 minutes on a warm build to populate KOCACHE, which is phenomenal, and then just ~90 seconds with KOCACHE populated. I just love ko. <3

@imjasonh imjasonh mentioned this issue Feb 2, 2022
Merged
@jdolitsky
Copy link
Author

Seems to work now! Thank you for such a quick turnaround.

Will re-open if it appears to happen again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add a mutex to map writing when using KOCACHE imjasonh/ko
2 participants
@imjasonh @jdolitsky