x/build: set up a qemu builder

[gopherbot]

by [email protected]:

I noticed a significant improvements in the arm tool chain :-)
I think it makes now a sense to take a closer look at bugs
in different environments.

The first shot is a qemu in user mode:
qemu-arm version 0.12.5
The host is (ubuntu 10.10)
GOARCH=arm
GORUN=qemu-arm
hg identify ba58b167f1fc+ tip

I tried with disabled peep or with GOARM=5, the bugs are similar.

Some syscalls are not covered, (I would propose
a QUEMU_UM to disable tests that use them):

(epoll_create)
        netchan\
        rpc\
        websocket\

Some packages show a non-deterministic behaviour,
a few do always break. I have no clue if qemu, go or
both are buggy:

        go/printer\
        compress/flate\
        compress/gzip\
        compress/zlib\
        os\
        big\
        rpc/jsonrpc\
        sync\ #(+ run.bash)
        bytes\

After disabling them, the test-suite *passed* all remaining tests.

----

Script bugs:
        run.bash: cmd/ebnflint ???
        run.bash: doc/progs [ "$GOHOSTOS" == windows ] ???
        run.bash: doc/codelab/wiki

The test/run script fails with:
./run -v
sed: can't read ./tmp.go: No such file or directory
fail: chan/powser2.go
fail: chan/select3.go
109a110,119
> =========== chan/powser2.go
> throw: malloc/free - deadlock
> 
> [signal 0xb code=0x1 addr=0x0 pc=0x15720]
> 
> 
> =========== chan/select3.go
> qemu-arm: /build/buildd/qemu-kvm-0.12.5+noroms/cpu-exec.c:784: handle_cpu_signal:
Assertion `__ret == (abi_ulong)__ret' failed.
> $RUNFILE: line 1: PID Aborted                 $E ./$A.out
> 
158a169,174
> 
> =========== fixedbugs/bug243.go
> Unsupported setsockopt level=41 optname=26 
> qemu: Unsupported syscall: 250
> Start pollServer: epoll_create: function not implemented
> Unsupported setsockopt level=41 optname=26 
1 known bugs; 0 unexpected bugs; test output differs


The captured outputs are attached.

Attachments:

1. qemu-um-bugs.txt (26808 bytes)

[gopherbot]

Comment 1 by [email protected]:

I forgot to clean up the environment. It was run
with GOMAXPROCS=2.
With GOMAXPROCS=1 everything passes except the 3 epoll
cases
        netchan\
        rpc\
        websocket\
and the os test
5g -o _gotest_.5 dir_linux.go error.go env.go exec.go file.go getwd.go path.go proc.go
stat_linux.go time.go types.go  env_unix.go file_unix.go sys_linux.go exec_unix.go  
rm -f _test/os.a
gopack grc _test/os.a _gotest_.5 
--- FAIL: os_test.TestSeek (0.0 seconds)
    #6: Seek(8589934592, 0) = 0, <nil> want 8589934592, nil
    #7: Seek(8589934592, 2) = 13, <nil> want 8589934605, nil
FAIL

[gopherbot]

Comment 2 by [email protected]:

I also forgot to unset DISABLE_NET_TESTS,
so all the non-working tests are (obviously):
        net\
        netchan\
        rpc\
        websocket\
        http\
        syslog\
        os\
----
Non-working benchmarks scripts:
cd pkg && make bench
...
...
cd ../../test/garbage && make bench
make[1]: Entering directory `/data4/soft/go/go/test/garbage'
5g parser.go stats.go
5l -o parser.out parser.5
./parser.out
./parser.out: 1: Syntax error: word unexpected (expecting ")")
make[1]: *** [parser.bench] Error 2
rm parser.5
make[1]: Leaving directory `/data4/soft/go/go/test/garbage'
make: *** [../../test/garbage.bench] Error 2
and
test/bench/timing.sh

[rsc]

Comment 3:

I run using the user-mode qemu all the time, but I use
the kernel binfmt misc support instead of setting $GORUN.
With DISABLE_NET_TESTS=1 all.bash runs to completion
on my linux/amd64 system.
The benchmark script looks like it just needs to use $GORUN.

Owner changed to [email protected].

Status changed to HelpWanted.

[gopherbot]

Comment 4 by [email protected]:

Thanks, indeed the binfmt misc support works much better. With GOMAXPROCS=1
and DISABLE_NET_TESTS=1 it passed all the tests.

[rsc]

Comment 5:

If you'd like to prepare a patch to insert $GORUN where it's needed, feel free.

[gopherbot]

Comment 6 by [email protected]:

I apologize, I was wrong, in the comment #4 the tests wasn't enabled.
Actually I can't found any qemu user-mode version that supports
epoll_create. How did you setup the binfmt?

[gopherbot]

Comment 7 by [email protected]:

http://code.google.com/p/go/source/detail?r=231c6b6747
almost fixes the GOMAXPROCS>1 random issues in comment #0
The success rate is much better, but not 100%.
The pkg/go/printer test breaks always:
throw: malloc/free - deadlock

Attachments:

1. go-qemu-arm-mp2.txt (16819 bytes)

[gopherbot]

Comment 8 by [email protected]:

Labels changed: added arch-arm.

[rsc]

Comment 9:

Labels changed: added priority-low, removed priority-medium.

[rsc]

Comment 10:

Labels changed: added priority-someday, removed priority-low.

[minux]

Comment 11:

Issue #5384 has been merged into this issue.

[rsc]

Comment 12:

Labels changed: added repo-main.

[rsc]

Comment 13:

Labels changed: added release-none.

[bcmills]

If we expect tests to pass in this configuration, then it should have a builder.
(See http://golang.org/wiki/PortingPolicy#requirements-for-a-new-port.)

If this configuration is sufficiently exotic that the tests don't pass, but nobody cares enough about it to set up (or run) a builder, then I don't think we should hold an issue open for it.

(CC @bradfitz @dmitshur @toothrot @cagedmantis)

[bradfitz]

I've historically ignored qemu-user because it doesn't quite test reality, but it might be worth trying to see what its state is, and it might be a good way to run a number of slower builders a bit faster. We've tried cross-compiling from fast GCE amd64 instances in the past and then just running tests sharded out over either physical non-x86 hardware or qemu system instances, but then we always hit the problem of how many tests of ours invoke go build themselves, and we end up compiling inside the slow environment instead.

Having qemu-user builders might mean we get a good compromise between fast host doing compilations and getting good enough arch emulation.

I think it's worth keeping this open for somebody to investigate.

[gopherbot]

Change https://golang.org/cl/206137 mentions this issue: cmd/go: delete flaky TestQEMUUserMode

[prattmic]

We've had several QEMU-only bugs recently (e.g., #68976, #69255).

QEMU is effectively a secondary implementation of the architecture when performing instruction emulation.

QEMU is also effectively a secondary implementation of Linux when running in "user mode" (similar to WSL1).

It would be nice to have an advisory builder to provide some warning of failures. There are a lot of variables: QEMU version, user mode or not, host arch, target arch. #68976 affects only user mode, and only older versions of QEMU, not the latest version.

Covering all of these configurations is too much. We should pick one that we think is one of the most common. Based on #68976 and #69255, that seems to be: QEMU version shipped with Debian, user mode. Targeting arm64 from host amd64 or vice versa both seem common; it's unclear which is more common, though I imagine amd64 host is easier to set up.

[zekth]

@prattmic not sure it helps but found a working case with QEMU: #69255 (comment)