runtime: running Go code on OpenBSD gomote fails when not running as root

[ianlancetaylor]

I don't know what is going on here, but recording since something is wrong.

When I use gomote run with the openbsd-amd64-62 gomote, everything works as expected. When I use gomote ssh to ssh into the gomote, the go tool consistently fails with the following stack trace.

The only obvious difference is that gomote run runs as root but gomote ssh does not.

CC @mknyszek @aclements @bradfitz

fatal error: failed to reserve page bitmap memory

runtime stack:
runtime.throw(0xa4504f, 0x24)
        /tmp/workdir/go/src/runtime/panic.go:1106 +0x72 fp=0x7f7ffffd5018 sp=0x7f7ffffd4fe8 pc=0x4331a2
runtime.(*pageAlloc).init(0xeb7b88, 0xeb7b80, 0xecff58)
        /tmp/workdir/go/src/runtime/mpagealloc.go:239 +0x162 fp=0x7f7ffffd5060 sp=0x7f7ffffd5018 pc=0x428c12
runtime.(*mheap).init(0xeb7b80)
        /tmp/workdir/go/src/runtime/mheap.go:694 +0x274 fp=0x7f7ffffd5088 sp=0x7f7ffffd5060 pc=0x425de4
runtime.mallocinit()
        /tmp/workdir/go/src/runtime/malloc.go:471 +0xff fp=0x7f7ffffd50b8 sp=0x7f7ffffd5088 pc=0x40c5af
runtime.schedinit()
        /tmp/workdir/go/src/runtime/proc.go:545 +0x60 fp=0x7f7ffffd5110 sp=0x7f7ffffd50b8 pc=0x436700
runtime.rt0_go(0x7f7ffffd5148, 0x1, 0x7f7ffffd5148, 0x0, 0x0, 0x1, 0x7f7ffffd5238, 0x0, 0x7f7ffffd523b, 0x7f7ffffd5254, ...)
        /tmp/workdir/go/src/runtime/asm_amd64.s:214 +0x125 fp=0x7f7ffffd5118 sp=0x7f7ffffd5110 pc=0x45eff5

[mknyszek]

If I were to guess, there's an RLIMIT_AS set for non-root users or something. I'll look into this now.

[bradfitz]

People who like this bug also like #10719.

/cc @bcmills

[ianlancetaylor]

For root bash's ulimit -v reports 33562624. For non-root, it reports 1576960.

[mknyszek]

@ianlancetaylor I investigated this with someone who knows OpenBSD a bit.

The number that's checked in the kernel from a PROT_NONE anonymous mapping is RLIMIT_DATA, which is limited for non-root users in login.conf. We can fix this on our builders by making datasize-cur and datasize-max unlimited, but if OpenBSD has a low default then that's a problem since Go no longer works out of the box on a newly-installed OpenBSD image.

Perhaps there's a workaround here but I need to give it some thought.

It's a little bit weird to me that a PROT_NONE mapping counts toward this on any platform. Linux does this too, but its default for everyone for virtual address space is unlimited and not 768 MiB.

[ianlancetaylor]

Just a note that the same problem still happens on OpenBSD 6.4.

[bradfitz]

/cc @mdempsky as FYI just because he likes OpenBSD issues.

[mdempsky]

It's been a while since I've looked at OpenBSD's mmap implementation. I seem to recall it took the stance that PROT_NONE mappings still mapped data (and thus counts towards RLIMIT_DATA), just without read/write access.

I see that http://cvsweb.openbsd.org/cgi-bin/cvsweb/ports/lang/go/Makefile?rev=1.75 uses ulimit -d $(ulimit -H -d) to raise the RLIMIT_DATA soft limit to the hard limit. But maybe that only works because ports builders usually have login class "staff" or "pbuild" (which set datasize-max=infinity), whereas gomote ssh is just giving a default login class (with datasize-max=768M)?

/cc @4a6f656c

[gopherbot]

Change https://golang.org/cl/207497 mentions this issue: runtime: convert page allocator bitmap to sparse array

[4a6f656c]

The login class will be the reason why the allocation worked when run as root, but not when run as a normal user. However, even when a user has datasize-max=infinity (and increases the data size soft limit), there is still an upper bound based on the amount of memory available to the machine - as such, on a laptop with 4GB of RAM, the ~8.5GB PROT_NONE allocation was still failing, even if run as a user with a login class of staff (or as root). It presumably only worked on the builders as root due to a significant amount of memory being available.

The same failure could also be triggered under Linux via ulimit -v (some code removed around 2017 had a comment noting that sysReserve could fail on 64-bit systems, either due to kernel enforced constraints or ulimit -v).

Obviously this was a pretty significant regression from Go 1.13 - I can confirm that with the sparse array change, I can once again build Go with a data size limit of 2GB (although there appears to have been another regression with the memory requirements for compiling cmd/compile/internal/ssa - 1.5GB is sufficient for Go 1.13.4) and run a Go binary with a data size limit of 768MB.

[ianlancetaylor]

@mknyszek Is there anything else to do on this issue or do we think that it is fixed?

[mknyszek]

@4a6f656c It's unfortunate that virtual address space is limited in this way, but c'est l'vie. I'm glad the sparse array change helped.

We were also aware of ulimit -v failure mode on Linux for a while. The default is infinity because virtual address space is cheap (for example, I can make a 2 TiB PROT_NONE mapping without issue on Linux, with either default and strict overcommit rules set). Generally speaking ulimit -v is not the most accurate way to limit physical memory use (it's only per-process!), and cgroups do much better, so I don't think folks tend to use it much anymore.

@ianlancetaylor I don't think there's anything else we want to do now in terms of reducing PROT_NONE memory mapped. We tried a few things and they were either quite complicated or had other problems.