runtime,cmd/compile: frequent memory corruption on NetBSD and OpenBSD since 2021-10-11 #49209

bcmills · 2021-10-28T21:19:53Z

greplogs --dashboard -md -l -e 'freeIndex is not valid' --since=2021-05-01

The text was updated successfully, but these errors were encountered:

bcmills · 2021-10-28T21:21:15Z

CC @bsiegert @coypoop

bcmills · 2021-10-28T21:23:31Z

This appears to be a regression in Go 1.18.

Since NetBSD is not a first-class port, this doesn't necessarily block the 1.18 release — however, if the regression remains at the time of the release it at least needs a clear writeup in the release notes. (That part, at least, is a release-blocker.)

bcmills · 2021-10-28T21:24:40Z

@jeremyfaller, any idea who on the Go team might have insight into what broke on the 18th specifically?

coypoop · 2021-10-29T04:08:16Z

I'm not sure this error is entirely a regression, as I've seen this with pre-Go-1.18 on NetBSD. But perhaps something is making it much more frequent.

jeremyfaller · 2021-10-29T19:35:29Z

Did a little digging on Oct17-18, and didn't see much that looked suspicious.

pinging @mknyszek

bcmills · 2021-11-02T15:56:18Z

There have been a lot of other failures since then that look like arbitrary memory corruption in the runtime.
Whatever this is, it looks like it's got all of the NetBSD builders in pretty bad shape.
(Silver lining: the failure rate is high enough that this should be feasible to bisect!)

greplogs --dashboard -md -l -e '(?ms)\Anetbsd.*fatal error:' --since=2021-10-01

mknyszek · 2021-11-08T23:11:25Z

On it.

mknyszek · 2021-11-09T00:54:34Z

goswarm $ GOROOT=$HOME/toolchain ./goswarm -match 'freeIndex is not valid' netbsd-386-9_0 go/src/all.bash
2021/11/09 00:09:59 Created instance user-mknyszek-netbsd-386-9_0-1...
2021/11/09 00:10:02 Created instance user-mknyszek-netbsd-386-9_0-2...
2021/11/09 00:10:03 Created instance user-mknyszek-netbsd-386-9_0-3...
2021/11/09 00:10:03 Created instance user-mknyszek-netbsd-386-9_0-4...
2021/11/09 00:10:05 Created instance user-mknyszek-netbsd-386-9_0-5...
2021/11/09 00:10:05 Created instance user-mknyszek-netbsd-386-9_0-6...
2021/11/09 00:10:07 Created instance user-mknyszek-netbsd-386-9_0-7...
2021/11/09 00:10:13 Pushed to user-mknyszek-netbsd-386-9_0-1.
2021/11/09 00:10:13 Running command on user-mknyszek-netbsd-386-9_0-1.
2021/11/09 00:10:15 Created instance user-mknyszek-netbsd-386-9_0-8...
2021/11/09 00:10:15 Created instance user-mknyszek-netbsd-386-9_0-9...
2021/11/09 00:10:15 Pushed to user-mknyszek-netbsd-386-9_0-2.
2021/11/09 00:10:15 Running command on user-mknyszek-netbsd-386-9_0-2.
2021/11/09 00:10:17 Pushed to user-mknyszek-netbsd-386-9_0-3.
2021/11/09 00:10:17 Running command on user-mknyszek-netbsd-386-9_0-3.
2021/11/09 00:10:17 Pushed to user-mknyszek-netbsd-386-9_0-4.
2021/11/09 00:10:17 Running command on user-mknyszek-netbsd-386-9_0-4.
2021/11/09 00:10:19 Pushed to user-mknyszek-netbsd-386-9_0-6.
2021/11/09 00:10:19 Running command on user-mknyszek-netbsd-386-9_0-6.
2021/11/09 00:10:19 Pushed to user-mknyszek-netbsd-386-9_0-5.
2021/11/09 00:10:19 Running command on user-mknyszek-netbsd-386-9_0-5.
2021/11/09 00:10:20 Pushed to user-mknyszek-netbsd-386-9_0-7.
2021/11/09 00:10:20 Running command on user-mknyszek-netbsd-386-9_0-7.
2021/11/09 00:10:28 Pushed to user-mknyszek-netbsd-386-9_0-8.
2021/11/09 00:10:28 Running command on user-mknyszek-netbsd-386-9_0-8.
2021/11/09 00:10:28 Pushed to user-mknyszek-netbsd-386-9_0-9.
2021/11/09 00:10:28 Running command on user-mknyszek-netbsd-386-9_0-9.
2021/11/09 00:10:29 Created instance user-mknyszek-netbsd-386-9_0-10...
2021/11/09 00:10:42 Pushed to user-mknyszek-netbsd-386-9_0-10.
2021/11/09 00:10:42 Running command on user-mknyszek-netbsd-386-9_0-10.
2021/11/09 00:21:43 Running command on user-mknyszek-netbsd-386-9_0-2.
2021/11/09 00:31:28 Running command on user-mknyszek-netbsd-386-9_0-1.
2021/11/09 00:31:29 Running command on user-mknyszek-netbsd-386-9_0-6.
2021/11/09 00:31:33 Running command on user-mknyszek-netbsd-386-9_0-8.
2021/11/09 00:31:54 Running command on user-mknyszek-netbsd-386-9_0-7.
2021/11/09 00:31:58 Running command on user-mknyszek-netbsd-386-9_0-3.
2021/11/09 00:31:58 Running command on user-mknyszek-netbsd-386-9_0-4.
2021/11/09 00:32:38 Running command on user-mknyszek-netbsd-386-9_0-10.
2021/11/09 00:33:05 Running command on user-mknyszek-netbsd-386-9_0-9.
2021/11/09 00:40:36 Running command on user-mknyszek-netbsd-386-9_0-5.
2021/11/09 00:42:10 Running command on user-mknyszek-netbsd-386-9_0-2.
2021/11/09 00:52:34 Running command on user-mknyszek-netbsd-386-9_0-1.
2021/11/09 00:52:37 Running command on user-mknyszek-netbsd-386-9_0-6.
2021/11/09 00:53:15 Running command on user-mknyszek-netbsd-386-9_0-7.
2021/11/09 00:53:18 Running command on user-mknyszek-netbsd-386-9_0-3.
2021/11/09 00:53:36 Running command on user-mknyszek-netbsd-386-9_0-4.

This is more difficult to reproduce than I anticipated...

mknyszek · 2021-11-09T14:27:09Z

My gomote swarm ran all night and I got nowhere, unfortunately.

bcmills · 2021-11-09T20:02:41Z

Looking more carefully at the fatal error failures, I notice that some of them are actually on the release branches.

For example, the 2021-10-28T18:48:25-e7eb6f6-9004433/netbsd-386-9_0 failure is on release-branch.go1.16.

@golang/release, was there a change to the builder images on or around Oct. 11 that could explain these failures? (If so, is there an older image we can restore, or a platform expert who can, say, try bisecting the kernel?)

bcmills · 2021-11-09T20:03:49Z

@mknyszek, the freeIndex is not valid failure mode seems to be rarer than some of the other failure modes. Would it make sense to cast a wider net for fatal errors that lead to core files?

mknyszek · 2021-11-09T20:19:29Z

Ah OK. I'll focus on other failure modes.

bcmills · 2021-11-09T20:20:53Z

Going back further in the history seems to show:

There was an interval with an extremely high netbsd failure rate around 2021-02-19; looks like it was runtime: fix over-eager SPWRITE check #44269.
Then there were a few intermittent failures with memory-corruption signatures — maybe once a month, mostly involving systemstack and/or wirep (maybe related to runtime: OpenBSD wirep related failures #49259?) — from 2021-03-10 until the onset of the unlock2 crashes (runtime: frequent crashes in unlock2 on NetBSD and OpenBSD since 2021-10-07 #49453), which started around 2021-09-22 on OpenBSD and first appeared 2021-10-07 on NetBSD.
Then a large uptick in failures in mid-October — maybe corresponding to a builder upgrade, or maybe even just the opening of the tree in October?

greplogs --dashboard -md -l -e '(?ms)\Anetbsd-(?:amd64|386).*fatal error:' --since=2021-01-01

The freeIndex is not valid failures have followed a similar trajectory to the memory-corruption failures in general, but have a much crisper starting point: 2021-10-18.

greplogs --dashboard -md -l -e 'freeIndex is not valid' --since=2021-01-01

mknyszek · 2021-11-16T02:11:39Z

Got a core dump... for a go1.4 failure. I feel like I'm cursed.

Building Go cmd/dist using /tmp/workdir/go1.4. (devel +0b3b511 Tue Dec 5 18:44:31 2017 +0000 netbsd/386)
Building Go toolchain1 using /tmp/workdir/go1.4.
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x1f8 pc=0x80553c7]

runtime stack:
runtime.throw(0x8428440, 0x2a)
	/home/bradfitz/go-netbsd-386-bootstrap/src/runtime/panic.go:616 +0x6b fp=0x1893bec8 sp=0x1893bebc pc=0x806e24b
runtime.sigpanic()
	/home/bradfitz/go-netbsd-386-bootstrap/src/runtime/signal_unix.go:366 +0x230 fp=0x1893beec sp=0x1893bec8 pc=0x8081480
runtime.unlock(0x86290b0)
	/home/bradfitz/go-netbsd-386-bootstrap/src/runtime/lock_sema.go:107 +0x97 fp=0x1893bf08 sp=0x1893beec pc=0x80553c7
runtime.goschedImpl(0x18960fc0)
	/home/bradfitz/go-netbsd-386-bootstrap/src/runtime/proc.go:2606 +0xcb fp=0x1893bf18 sp=0x1893bf08 pc=0x8074c7b
runtime.gopreempt_m(0x18960fc0)
	/home/bradfitz/go-netbsd-386-bootstrap/src/runtime/proc.go:2636 +0x26 fp=0x1893bf20 sp=0x1893bf18 pc=0x8074d36
runtime.newstack()
	/home/bradfitz/go-netbsd-386-bootstrap/src/runtime/stack.go:1039 +0x213 fp=0x1893bfd8 sp=0x1893bf20 pc=0x8084393
runtime.morestack()
	/home/bradfitz/go-netbsd-386-bootstrap/src/runtime/asm_386.s:536 +0x70 fp=0x1893bfdc sp=0x1893bfd8 pc=0x8092b50

mknyszek · 2021-11-16T03:09:28Z

Got 2 crashes in one go: a freeIndex crash (!) and a scanstack - bad status crash. I also got a whole bunch of cores.

mknyszek · 2021-11-16T04:31:24Z

OK so I can't seem to find the core for the scanstack issue, but the freeIndex issue is kind of interesting. First let me say that I'm pretty confident nothing is off here -- all the pointers line up with what's in the stack trace, nothing's inconsistent so far.

So, the check that's failing has code that looks like

 871     s = c.alloc[spc]
 872     shouldhelpgc = false
 873     freeIndex := s.nextFreeIndex()
 874     if freeIndex == s.nelems {
 875         // The span is full.
 876         if uintptr(s.allocCount) != s.nelems {
 877             println("runtime: s.allocCount=", s.allocCount, "s.nelems=", s.nelems)
 878             throw("s.allocCount != s.nelems && freeIndex == s.nelems")
 879         }
 880         c.refill(spc)
 881         shouldhelpgc = true
 882         s = c.alloc[spc]                                                                     
 883
 884         freeIndex = s.nextFreeIndex()
 885     }
 886
 887     if freeIndex >= s.nelems {
 888         throw("freeIndex is not valid")
 889     }

I accessed the relevant mspan in the core dump by: finding the crashing goroutine by goid, looking up its p, accessing that p's mcache, and then picking out the span class from the backtrace (note that this is reliable because that value never changes for the duration of nextFree, also it lines up with the size passed to mallocgc: span class 0x49 (size class 36) for size 0x600). Note also that s.nextFreeIndex always writes to mspan.freeindex before returning.

When I looked at s.nelems and s.freeindex, I found the values 5 and 0 respectively. That should not be causing the check to fire.

If all this makes sense, then I think that means what we have here is some kind of stack corruption. Specifically, the value of freeIndex on the stack is getting corrupted. Unfortunately I can't figure out how to coax gdb into letting me inspect an off-thread stack and get into that frame to print locals. That would confirm it for sure.

bcmills · 2021-11-17T17:39:56Z

Looks like this also affects OpenBSD:

greplogs --dashboard -md -l -e '(?ms)\Aopenbsd-.*freeIndex is not valid'

2021-11-17T04:31:22-f384c70/openbsd-386-70
2021-11-15T23:55:09-9e13a88/openbsd-386-70
2021-11-10T19:35:55-03971e3-b954f58/openbsd-386-70
2021-11-02T19:37:42-1011e26/openbsd-386-68

cherrymui · 2021-12-15T18:42:44Z

From #34988 it seems we can conclude that it is a kernel issue on NetBSD? Is there any conclusion for OpenBSD?

Do we still need to keep this issue open? Thanks.

gopherbot · 2021-12-15T18:47:45Z

Change https://golang.org/cl/372355 mentions this issue: doc/go1.18: document OpenBSD & NetBSD memory corruption

bcmills · 2021-12-15T18:50:43Z

Do we still need to keep this issue open?

From my perspective, we do still need to keep the issue open, to the extent that we need to:

document the architectures on which Go is known not to work, and/or
configure the Go project's builders to work around the kernel issue (e.g. by avoiding running on the affected AMD CPUs), and/or
explicitly drop support for the affected ports until such time as the port maintainers can fix or work around the upstream bugs.

bcmills · 2021-12-15T18:52:28Z

Hmm. Actually, I think it suffices to mark this as a duplicate of #34988, and do the above things for that bug. 👍

prattmic · 2021-12-15T19:11:11Z

Is there any conclusion for OpenBSD?

Not yet, as the C reproducer doesn't currently work on OpenBSD. We probably need to go back to David's Go reproducer and see if that works, and if so try to get it working in C again.

cagedmantis · 2022-01-07T14:32:42Z

This is not a bug in Go. The failing builders will be annotated with a known issue until it is resolved. Because of this, it is no longer a release blocker.

gopherbot · 2022-01-11T17:20:07Z

Change https://golang.org/cl/377474 mentions this issue: dashboard: clean up builders affected by memory corruption

Remove freebsd 12.2, which is replaced by 12.3 with the XSAVE fix. Move freebsd 11.* to N2 machines, which are not affected. Remove openbsd and netbsd e2/n1/n2/n2d-specific configurations now that we have mostly understood the nature of that problem. Keep one around so that the runtime team can create gomotes. Move the "official" builder to the n2 cpu that works. For golang/go#49967, golang/go#49209, golang/go#40561. Fixes golang/go#50496. Change-Id: If6989317f06cbec95d5addb19d9e968aecfa3f8a Reviewed-on: https://go-review.googlesource.com/c/build/+/377474 Trust: Heschi Kreinick <[email protected]> Run-TryBot: Heschi Kreinick <[email protected]> Reviewed-by: Carlos Amedee <[email protected]> Reviewed-by: Alex Rakoczy <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]> TryBot-Result: Gopher Robot <[email protected]>

Knowing whether test failures are correlated with specific CPU models on has proven useful on several issues. Log it for prior to testing so it is always available. internal/sysinfo provides the CPU model, but it is not available in the bootstrap toolchain, so we can't access this unconditionally in cmd/dist. Instead use a build-tagged file, as the final version of cmd/dist will use the final toolchain. The addition of new data to the beginning of cmd/dist output will break x/build/cmd/coordinator's banner parsing, leaving extra lines in the log output, though information will not be lost. https://golang.org/cl/372538 fixes up the coordinator and should be submitted and deployed before this CL is submitted. For #46272. For #49209. For #50146. Change-Id: I515d2ec58e4c0034b76bf624ecaab38f16146074 Reviewed-on: https://go-review.googlesource.com/c/go/+/371474 Trust: Benny Siegert <[email protected]> Reviewed-by: Benny Siegert <[email protected]> Trust: Michael Pratt <[email protected]> Run-TryBot: Michael Pratt <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Reviewed-by: Austin Clements <[email protected]>

gopherbot · 2022-01-14T21:15:53Z

Change https://golang.org/cl/378589 mentions this issue: cmd/dist: log CPU model when testing

ianlancetaylor · 2022-01-29T00:29:10Z

If our conclusion is that this is not a bug in Go, we should consider moving this to the 1.19 milestone.

bcmills · 2022-01-29T00:32:25Z

The issue is documented and the builders are configured to work around it. As far as I am concerned it would be appropriate to close this issue.

ianlancetaylor · 2022-01-29T01:04:16Z

SGTM. Thanks.

Knowing whether test failures are correlated with specific CPU models on has proven useful on several issues. Log it for prior to testing so it is always available. internal/sysinfo provides the CPU model, but it is not available in the bootstrap toolchain, so we can't access this in cmd/dist. Instead use a separate binary which cmd/dist will only build once testing begins. The addition of new data to the beginning of cmd/dist output will break x/build/cmd/coordinator's banner parsing, leaving extra lines in the log output, though information will not be lost. https://golang.org/cl/372538 fixes up the coordinator and should be submitted and deployed before this CL is submitted. This is a redo of CL 371474. It switches back to the original approach of using a separate binary, as the bootstap toolchain won't allow cmd/dist to import internal packages. For #46272. For #49209. For #50146. Change-Id: I906bbda987902a2120c5183290a4e89a2440de58 Reviewed-on: https://go-review.googlesource.com/c/go/+/378589 Reviewed-by: Austin Clements <[email protected]> Trust: Michael Pratt <[email protected]> Run-TryBot: Michael Pratt <[email protected]> TryBot-Result: Gopher Robot <[email protected]>

Knowing whether test failures are correlated with specific CPU models on has proven useful on several issues. Log it for prior to testing so it is always available. internal/sysinfo provides the CPU model, but it is not available in the bootstrap toolchain, so we can't access this unconditionally in cmd/dist. Instead use a build-tagged file, as the final version of cmd/dist will use the final toolchain. The addition of new data to the beginning of cmd/dist output will break x/build/cmd/coordinator's banner parsing, leaving extra lines in the log output, though information will not be lost. https://golang.org/cl/372538 fixes up the coordinator and should be submitted and deployed before this CL is submitted. For golang#46272. For golang#49209. For golang#50146. Change-Id: I515d2ec58e4c0034b76bf624ecaab38f16146074 Reviewed-on: https://go-review.googlesource.com/c/go/+/371474 Trust: Benny Siegert <[email protected]> Reviewed-by: Benny Siegert <[email protected]> Trust: Michael Pratt <[email protected]> Run-TryBot: Michael Pratt <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Reviewed-by: Austin Clements <[email protected]>

bcmills changed the title ~~runtime,cmd/compile: frequent failures with "freeIndex is not valid" since 2021-10-18~~ runtime,cmd/compile: frequent failures with "freeIndex is not valid" on NetBSD since 2021-10-18 Oct 28, 2021

bcmills added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker labels Oct 28, 2021

bcmills added this to the Go1.18 milestone Oct 28, 2021

bcmills added the OS-NetBSD label Oct 28, 2021

bcmills changed the title ~~runtime,cmd/compile: frequent failures with "freeIndex is not valid" on NetBSD since 2021-10-18~~ runtime,cmd/compile: frequent memory corruption on NetBSD since 2021-10-11 Nov 2, 2021

This was referenced Nov 5, 2021

runtime: apparent deadlock in TestNoShrinkStackWhileParking on netbsd-arm64-bsiegert #49382

Open

runtime: frequent crashes in unlock2 on NetBSD and OpenBSD since 2021-10-07 #49453

Closed

mknyszek self-assigned this Nov 8, 2021

bcmills mentioned this issue Nov 9, 2021

os: on NetBSD, *Process.Wait sometimes deadlocks after cmd.Process.Signal returns "process already finished" #48789

Closed

bcmills mentioned this issue Nov 10, 2021

x/tools/internal/lsp/source: test failures with diffs on "//@complete" lines #49463

Closed

bcmills changed the title ~~runtime,cmd/compile: frequent memory corruption on NetBSD since 2021-10-11~~ runtime,cmd/compile: frequent memory corruption on NetBSD and OpenBSD since 2021-10-11 Nov 17, 2021

toothrot added the okay-after-beta1 Used by release team to mark a release-blocker issue as okay to resolve either before or after beta1 label Nov 19, 2021

This was referenced Dec 14, 2021

net: TestVariousDeadlines failures with "client stuck in Dial+Copy" on NetBSD builders #37883

Closed

net: *TCPConn deadlines missed by more than 25% on NetBSD and OpenBSD #50189

Open

cagedmantis removed the release-blocker label Jan 5, 2022

bcmills mentioned this issue Jan 7, 2022

x/build: switch NetBSD and OpenBSD TryBots to builders that are not affected by known issues #50496

Closed

This was referenced Jan 12, 2022

runtime: "schedule: holding locks" #44544

Closed

x/tools/gopls/internal/regtest/workspace: failures with ‘Unmet: completed work "diagnosing initial workspace load"’ #50582

Open

heschi mentioned this issue Jan 12, 2022

x/build/internal/releasetargets: evaluate for each upcoming major Go release #40561

Open

This was referenced Jan 24, 2022

x/tools/gopls/internal/regtest/codelens: TestGCDetails timeouts with non-empty outstanding work #49902

Closed

x/build/cmd/coordinator: failures are sometimes missing error output #39349

Closed

ianlancetaylor mentioned this issue Jan 29, 2022

runtime: pollCache.free crash #49273

Closed

ianlancetaylor closed this as completed Jan 29, 2022

jmpesp mentioned this issue Jun 8, 2022

runtime: frequent GC related asserts seen on illumos based OSes (but only on AMD!) #53289

Open

rsc unassigned mknyszek Jun 23, 2022

golang locked and limited conversation to collaborators Jun 23, 2023

gopherbot added the FrozenDueToAge label Jun 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runtime,cmd/compile: frequent memory corruption on NetBSD and OpenBSD since 2021-10-11 #49209

runtime,cmd/compile: frequent memory corruption on NetBSD and OpenBSD since 2021-10-11 #49209

bcmills commented Oct 28, 2021 •

edited

Loading

bcmills commented Oct 28, 2021

bcmills commented Oct 28, 2021 •

edited

Loading

bcmills commented Oct 28, 2021

coypoop commented Oct 29, 2021

jeremyfaller commented Oct 29, 2021

bcmills commented Nov 2, 2021 •

edited

Loading

mknyszek commented Nov 8, 2021

mknyszek commented Nov 9, 2021

mknyszek commented Nov 9, 2021

bcmills commented Nov 9, 2021

bcmills commented Nov 9, 2021

mknyszek commented Nov 9, 2021

bcmills commented Nov 9, 2021

mknyszek commented Nov 16, 2021

mknyszek commented Nov 16, 2021

mknyszek commented Nov 16, 2021 •

edited

Loading

bcmills commented Nov 17, 2021

cherrymui commented Dec 15, 2021 •

edited

Loading

gopherbot commented Dec 15, 2021

bcmills commented Dec 15, 2021 •

edited

Loading

bcmills commented Dec 15, 2021

prattmic commented Dec 15, 2021

cagedmantis commented Jan 7, 2022

gopherbot commented Jan 11, 2022

gopherbot commented Jan 14, 2022

ianlancetaylor commented Jan 29, 2022

bcmills commented Jan 29, 2022

ianlancetaylor commented Jan 29, 2022

runtime,cmd/compile: frequent memory corruption on NetBSD and OpenBSD since 2021-10-11 #49209

runtime,cmd/compile: frequent memory corruption on NetBSD and OpenBSD since 2021-10-11 #49209

Comments

bcmills commented Oct 28, 2021 • edited Loading

bcmills commented Oct 28, 2021

bcmills commented Oct 28, 2021 • edited Loading

bcmills commented Oct 28, 2021

coypoop commented Oct 29, 2021

jeremyfaller commented Oct 29, 2021

bcmills commented Nov 2, 2021 • edited Loading

mknyszek commented Nov 8, 2021

mknyszek commented Nov 9, 2021

mknyszek commented Nov 9, 2021

bcmills commented Nov 9, 2021

bcmills commented Nov 9, 2021

mknyszek commented Nov 9, 2021

bcmills commented Nov 9, 2021

mknyszek commented Nov 16, 2021

mknyszek commented Nov 16, 2021

mknyszek commented Nov 16, 2021 • edited Loading

bcmills commented Nov 17, 2021

cherrymui commented Dec 15, 2021 • edited Loading

gopherbot commented Dec 15, 2021

bcmills commented Dec 15, 2021 • edited Loading

bcmills commented Dec 15, 2021

prattmic commented Dec 15, 2021

cagedmantis commented Jan 7, 2022

gopherbot commented Jan 11, 2022

gopherbot commented Jan 14, 2022

ianlancetaylor commented Jan 29, 2022

bcmills commented Jan 29, 2022

ianlancetaylor commented Jan 29, 2022

bcmills commented Oct 28, 2021 •

edited

Loading

bcmills commented Oct 28, 2021 •

edited

Loading

bcmills commented Nov 2, 2021 •

edited

Loading

mknyszek commented Nov 16, 2021 •

edited

Loading

cherrymui commented Dec 15, 2021 •

edited

Loading

bcmills commented Dec 15, 2021 •

edited

Loading