x/build: update NetBSD builders to NetBSD 8-beta #20852

bradfitz · 2017-06-29T19:13:52Z

In https://go-review.googlesource.com/c/47036/, Christos Zoulas mentions that there are known bugs in the NetBSD 7.1 kqueue code and we should switch our builders to NetBSD 8-beta.

Also mentioned in #20836 (comment)

This is a tracking bug to update our NetBSD builders past 7.1

I've updated https://golang.org/wiki/MinimumRequirements too.

/cc @krytarowski @bsiegert @zoulasc @adams-sarah @andybons

zoulasc · 2017-07-09T14:01:17Z

This should also be fixed in the 7.1 branch now, builds available from:
https://releng.netbsd.org/builds/netbsd-7-1/201707090640Z/

bsiegert · 2017-07-11T11:20:58Z

I would be more comfortable with a netbsd-8 system than with 7.1. http://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=52034 is still pending, and it means that significant disk activity is likely to crash the VM, or make it hang.

gopherbot · 2017-07-14T16:51:56Z

CL https://golang.org/cl/48091 mentions this issue.

There have been a number of fixes for issues found with the Go test suite in kqueue, signal handling and vio drivers. Updates golang/go#20852 Change-Id: I95861220990c05f3e89a4a40893e17758d29ebe0 Reviewed-on: https://go-review.googlesource.com/48091 Reviewed-by: Brad Fitzpatrick <[email protected]>

gopherbot · 2017-07-14T16:52:11Z

CL https://golang.org/cl/48552 mentions this issue.

bradfitz · 2017-07-14T17:10:57Z

@bsiegert, this doesn't boot on GCE as far as I can tell:

$ gomote create netbsd-amd64-8branch
Error running create: failed to create buildlet: 500 Internal Server Error: buildlet didn't come up at http://10.240.0.21 in 5m0s

Please let me know once you've actually run it on GCE.

Until that time, Go will continue to not run NetBSD builders.

bradfitz · 2017-07-14T17:21:51Z

I didn't mean to close this. Reopened.

zoulasc · 2017-07-15T16:49:52Z

Are there any log messages on how -8 fails to boot on GCE, or are there any GCE hosts available for us to try?

bradfitz · 2017-07-15T16:57:18Z

I didn't have time to capture the serial console. I had assumed @bsiegert tested this.

You can create small GCE VM instances yourself for free: https://cloud.google.com/free/

bsiegert · 2017-07-15T19:13:09Z

The relevant bit from the serial console:

+ curl -H 'Metadata-Flavor: Google' 'http://metadata.google.internal/computeMetadata/v1/instance/attributes/buildle
t-binary-url'
Shared object "libssl.so.10" not found
+ curl -o /buildlet
Shared object "libssl.so.10" not found
+ echo 'giving up'
+ command echo 'giving up'
giving up
+ sleep 10
+ halt -p
Jul 15 19:11:28 netbsd8-builder-20170715 halt: halted by root

The binary packages are for NetBSD-7, and I suppose they are linked against base OpenSSL from NetBSD-7 :( Sorry for not catching this before sending out the CL.

zoulasc · 2017-07-15T20:13:33Z

Yes, it is unfortunate that there are no pre-built packages for -8 yet. I will work on that. Do you need anything else from me?

bsiegert · 2017-07-17T09:25:54Z

I'll try unpacking a /lib and /usr/lib from NetBSD-7 to fix this.

zoulasc · 2017-07-17T23:26:50Z

the better way is to make the symlinks to the missing shared libraries to point to the new ones...

bsiegert · 2017-07-18T18:20:29Z

@zoulasc That sounds unsafe if there was a major bump in between.

gopherbot · 2017-07-18T19:20:00Z

CL https://golang.org/cl/49670 mentions this issue.

zoulasc · 2017-07-19T11:59:54Z

it is not an issue, people were just being conservative with libcrypto. OTOH if you replace libraries with older versions every binary on the system will break. But no worries, I'll have packages built soon.

gopherbot · 2017-08-03T14:46:50Z

Change https://golang.org/cl/53010 mentions this issue: x/build: use NetBSD-8 packages for netbsd-amd64 builder

This should fix the latest set of errors that bradfitz@ was seeing. Updates golang/go#20852 Change-Id: Iad7d6a479c5546395de907731643c1d817fe625d Reviewed-on: https://go-review.googlesource.com/53010 Reviewed-by: Brad Fitzpatrick <[email protected]>

bradfitz · 2017-08-04T04:22:23Z

@bsiegert, it still doesn't work.

Logs from the new x/build/cmd/debugnewvm tool running debugnewvm --host=host-netbsd-8branch --run-build=netbsd-amd64-8branch:

https://gist.github.com/bradfitz/3a6ba42ed3f9d7a47338e2172cfab9bb

bsiegert · 2017-08-04T09:06:34Z

Thanks for testing, and thanks for making me aware of debugnewvm.

It appears that this time, there has been a regression around setting routes. There is a default route via .0.1 but the host route to .0.1 is missing for some reason.

bsiegert · 2017-09-05T07:33:45Z

It took me a long time to find out, but it is a small bug in dhcpcd, the DHCP client. I will get this fix into NetBSD-8, then try again.

bsiegert · 2017-09-05T12:19:54Z

http://releng.netbsd.org/cgi-bin/req-8.cgi?show=266

bradfitz · 2017-12-01T01:21:09Z

@bsiegert, I just rebuilt the 386 image. It boots saying NetBSD 8.0_BETA (GENERIC.201711301510Z) which is newer than the 201711280120Z I see above. But still no luck:

virtio0: Virtio SCSI Device (rev. 0x00)
virtio0: no matching child driver; not configured

I'll try again tomorrow.

krytarowski · 2017-12-01T01:33:56Z

https://releng.netbsd.org/cgi-bin/req-8.cgi just monitor 413. Martin who committed the fix cannot pullup it and we need another person from releng to test and merge the patch.

bradfitz · 2017-12-01T01:35:50Z

@krytarowski, thanks for the info & link!

gopherbot · 2017-12-01T07:27:06Z

Change https://golang.org/cl/81356 mentions this issue: dashboard, cmd/buildlet, env/netbsd: enable netbsd ssh, re-disable netbsd-386

…tbsd-386 Fixes golang/go#22872 Updates golang/go#20852 Change-Id: I2b3454cb59a434cb12b10bb93cb4dd21d5187281 Reviewed-on: https://go-review.googlesource.com/81356 Reviewed-by: Andrew Bonventre <[email protected]>

gopherbot · 2017-12-04T04:06:00Z

Change https://golang.org/cl/81735 mentions this issue: dashboard: re-enable NetBSD/amd64 builder

Updates golang/go#22968 Updates golang/go#20852 Change-Id: Ida2bc7213d418735e897d0bae8e5fba9ae18fd6c Reviewed-on: https://go-review.googlesource.com/81735 Reviewed-by: Brad Fitzpatrick <[email protected]>

bradfitz · 2017-12-05T19:11:58Z

@bsiegert, the NetBSD/386 VM now boots on GCE, but I can't find any NetBSD mirror with binary packages for i386, which means we have no curl and we can't download the buildlet.

The builder script fails on these lines,

https://github.com/golang/build/blob/master/env/netbsd-amd64/mkvm.py#L63

But doesn't stop building the VM image on error.

localhost# PS1='anita-root-shell-pr''ompt-1512499919.93# '
PS1='anita-root-shell-pr''ompt-1512499919.93# '
anita-root-shell-prompt-1512499919.93# env PKG_PATH=http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/i386/8.0_2017Q2/All/ pkg_add bash curl
env PKG_PATH=http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/i386/8.0_2017Q2/All/ pkg_add bash curl
pkg_add: Can't process http://ftp.netbsd.org:80/pub/pkgsrc/packages/NetBSD/i386/8.0_2017Q2/All//bash*: Not Found
pkg_add: no pkg found for 'bash', sorry.
pkg_add: Can't process http://ftp.netbsd.org:80/pub/pkgsrc/packages/NetBSD/i386/8.0_2017Q2/All//curl*: Not Found
pkg_add: no pkg found for 'curl', sorry.
pkg_add: 2 package additions failed
anita-root-shell-prompt-1512499919.93# echo exit_status=$?=
echo exit_status=$?=
exit_status=1=
anita-root-shell-prompt-1512499919.93# exec /bin/sh
exec /bin/sh
localhost# PS1='anita-root-shell-pr''ompt-1512499922.04# '
PS1='anita-root-shell-pr''ompt-1512499922.04# '
anita-root-shell-prompt-1512499922.04# env PKG_PATH=http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/i386/8.0_2017Q2/All/ pkg_add git-base
env PKG_PATH=http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/i386/8.0_2017Q2/All/ pkg_add git-base
pkg_add: Can't process http://ftp.netbsd.org:80/pub/pkgsrc/packages/NetBSD/i386/8.0_2017Q2/All//git-base*: Not Found
pkg_add: no pkg found for 'git-base', sorry.
pkg_add: 1 package addition failed

Does somebody build i386 packages?

Or do we need to build from source? (That's what I did on my test NetBSD/i386 VM, but it was super tedious and involved checking out all of the pkgsrc cvs repo.... I'd rather not do that all in qemu)

krytarowski · 2017-12-05T19:27:17Z

We need to rebuild the packages for NetBSD-8/i386.

I've noted execution of "race" tests for FreeBSD, Linux, Windows - is this ThreadSanitizer (TSan) related? If so, we could add a NetBSD/amd64 target.

bradfitz · 2017-12-05T19:30:49Z

@krytarowski, who rebuilds packages? When can that be done by?

As for race detector support, that's unrelated to this issue. There is at least #19273 open for more platform support. Or you could file a new bug. But it would likely require upstream TSAN support. It won't happen for Go 1.10.

krytarowski · 2017-12-05T19:34:45Z

We, have already passed the requests internally to responsible developers.If I recall correctly, Manuel Bouyer was generating them.. however -current branches (like -8) are special cases so they will be generated on demand.

bradfitz · 2017-12-05T19:37:38Z

Oh, you already ported TSAN (https://blog.netbsd.org/tnf/entry/one_year_checkpoint_and_thread). Feel free to file a new bug for tracking Go support.

zoulasc · 2017-12-05T19:54:07Z

I am trying to build i386 packages for NetBSD-8.0...

gopherbot · 2017-12-05T23:40:39Z

Change https://golang.org/cl/82018 mentions this issue: dashboard: re-disable netbsd/386 builder until binary packages are built

Also update the bootstrap toolchain (verified in VMware VM locally). Updates golang/go#20852 Change-Id: Iac875419ed8dc51fb9db09d8948067f24c601e17 Reviewed-on: https://go-review.googlesource.com/82018 Reviewed-by: Andrew Bonventre <[email protected]>

zoulasc · 2017-12-06T04:01:18Z

There should be packages now for 8.0 + i386. Please let me know if I missed any dependencies.

bradfitz · 2017-12-06T04:06:28Z

@zoulasc, I don't seen mozilla-rootcerts here:

http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/i386/8.0_2017Q3/All/

The complete set we try to install is:

    "env PKG_PATH=http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/%s/%s/All/ pkg_add bash curl" % (arch, release),
    "env PKG_PATH=http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/%s/%s/All/ pkg_add git-base" % (arch, release),
    "env PKG_PATH=http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/%s/%s/All/ pkg_add mozilla-rootcerts go14" % (arch, release),
    # Interactive debugging tools for users using gomote ssh:
    "env PKG_PATH=http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/%s/%s/All/ pkg_add emacs25-nox11 vim screen" % (arch, release),

But emacs, vim, screen, go14, and even git-base should be optional. They're more nice-to-have for debugging. But without mozilla-rootcerts, curl & Go might have issues fetching resources.

zoulasc · 2017-12-06T04:07:51Z

stay tuned :-)

zoulasc · 2017-12-06T05:14:49Z

you should have everything but emacs... which should come later.

bradfitz · 2017-12-06T05:54:25Z

@zoulasc, thanks. I'm kicking off a build of the image.

zoulasc · 2017-12-06T05:55:11Z

thank you for all your patience :-) I am building git now.

gopherbot · 2017-12-06T06:22:21Z

Change https://golang.org/cl/82019 mentions this issue: dashboard, env/netbsd-386: update and re-enable netbsd/386 builder

The necessary binary packages are now available at: http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/i386/8.0_2017Q3/All/ Updates golang/go#20852 Change-Id: I20660a29f9e4044899de75d1b632c34619fca19b Reviewed-on: https://go-review.googlesource.com/82019 Reviewed-by: Brad Fitzpatrick <[email protected]>

bradfitz · 2017-12-06T06:26:33Z

Okay, deployed. I have it configured to only use 1 VM at a time so it won't use our self-imposed 500 VM quota all at once to catch up, but it'll catch up slowly on its own. I ran one test build manually and watched it get through the start, so I see networking on GCE now works, and bash & curl work, the buildlet starts (I needed to rebuild it for the sysctl fix), and make.bash starts running. I assume it'll be fine. Famous last words. Off to sleep...

Thanks again, @zoulasc and @bsiegert and @krytarowski!

krytarowski · 2017-12-06T17:38:00Z

Thanks for your effort! The next release should ship with TSan, MSan and ARM CI machines on NetBSD!

andybons · 2017-12-06T17:49:26Z

Really impressed with all the collaboration on this effort. Nice work, everyone! 🎉

bradfitz · 2017-12-06T17:51:10Z

And looks good this morning. Still some flakes here & there, but much better.

I'm going to close this bug. We can open new ones for specific issues that come up.

bradfitz added the OS-NetBSD label Jun 29, 2017

gopherbot added this to the Unreleased milestone Jun 29, 2017

gopherbot added the Builders x/build issues (builders, bots, dashboards) label Jun 29, 2017

bradfitz modified the milestones: Go1.10, Unreleased Jun 29, 2017

bradfitz added the help wanted label Jul 14, 2017

gopherbot closed this as completed in golang/build@d525484 Jul 14, 2017

bradfitz reopened this Jul 14, 2017

bradfitz closed this as completed Dec 6, 2017

golang locked and limited conversation to collaborators Dec 6, 2018

gopherbot added the FrozenDueToAge label Dec 6, 2018

x/build: update NetBSD builders to NetBSD 8-beta #20852

x/build: update NetBSD builders to NetBSD 8-beta #20852

Comments

bradfitz commented Jun 29, 2017

zoulasc commented Jul 9, 2017

bsiegert commented Jul 11, 2017

gopherbot commented Jul 14, 2017

gopherbot commented Jul 14, 2017

bradfitz commented Jul 14, 2017

bradfitz commented Jul 14, 2017

zoulasc commented Jul 15, 2017

bradfitz commented Jul 15, 2017

bsiegert commented Jul 15, 2017

zoulasc commented Jul 15, 2017

bsiegert commented Jul 17, 2017

zoulasc commented Jul 17, 2017

bsiegert commented Jul 18, 2017

gopherbot commented Jul 18, 2017

zoulasc commented Jul 19, 2017

gopherbot commented Aug 3, 2017

bradfitz commented Aug 4, 2017

bsiegert commented Aug 4, 2017

bsiegert commented Sep 5, 2017

bsiegert commented Sep 5, 2017

bradfitz commented Dec 1, 2017

krytarowski commented Dec 1, 2017

bradfitz commented Dec 1, 2017

gopherbot commented Dec 1, 2017

gopherbot commented Dec 4, 2017

bradfitz commented Dec 5, 2017

krytarowski commented Dec 5, 2017

bradfitz commented Dec 5, 2017

krytarowski commented Dec 5, 2017

bradfitz commented Dec 5, 2017

zoulasc commented Dec 5, 2017

gopherbot commented Dec 5, 2017

zoulasc commented Dec 6, 2017

bradfitz commented Dec 6, 2017 • edited Loading

zoulasc commented Dec 6, 2017

zoulasc commented Dec 6, 2017

bradfitz commented Dec 6, 2017

zoulasc commented Dec 6, 2017

gopherbot commented Dec 6, 2017

bradfitz commented Dec 6, 2017

krytarowski commented Dec 6, 2017 • edited Loading

andybons commented Dec 6, 2017

bradfitz commented Dec 6, 2017

bradfitz commented Dec 6, 2017 •

edited

Loading

krytarowski commented Dec 6, 2017 •

edited

Loading