Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aarch64: update bootstrap-files #151399

Merged
merged 1 commit into from
Jan 4, 2022

Conversation

dguibert
Copy link
Member

@dguibert dguibert commented Dec 20, 2021

This PR fixes #112086.

The bootstrap files have been updated from these jobs on hydra:

https://hydra.nixos.org/job/nixpkgs/trunk/stdenvBootstrapTools.aarch64-linux.dist ->
https://hydra.nixos.org/build/160330465#tabs-buildinputs

These commands have been run to get the output hashes as FODs:

nix store add-file --name bootstrap-tools.tar.xz  $(nix-store -r /nix/store/9y4qnwflnxkjrqnfbzhsjv6zq2r0m48z-stdenv-bootstrap-tools --option binary-caches https://cache.nixos.org)/on-server/bootstrap-tools.tar.xz
nix store add-path --name busybox                 $(nix-store -r /nix/store/9y4qnwflnxkjrqnfbzhsjv6zq2r0m48z-stdenv-bootstrap-tools --option binary-caches https://cache.nixos.org)/on-server/busybox
Motivation for this change
Things done
  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 22.05 Release Notes (or backporting 21.11 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
    • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
  • Fits CONTRIBUTING.md.

@github-actions github-actions bot added the 6.topic: stdenv Standard environment label Dec 20, 2021
@dguibert dguibert requested a review from vcunat December 20, 2021 09:20
@vcunat
Copy link
Member

vcunat commented Dec 20, 2021

Well, for aarch64-linux we have native builders on Hydra, so I expect we'd prefer to take the native bootstrap tools. (I didn't realize this difference when pointing you to risc-v.) https://hydra.nixos.org/job/nixpkgs/trunk/stdenvBootstrapTools.aarch64-linux.dist probably

@ofborg ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux labels Dec 20, 2021
fixes NixOS#112086

Fetch bootstrap files built by Hydra and re-add them as FODs to skip
currently dangling URLs. They are already available on
https://cache.nixos.org

https://hydra.nixos.org/job/nixpkgs/trunk/stdenvBootstrapTools.aarch64-linux.dist ->
https://hydra.nixos.org/build/160330465#tabs-buildinputs

These commands have been run to get the output hashes as FODs:

nix store add-file --name bootstrap-tools.tar.xz  $(nix-store -r /nix/store/9y4qnwflnxkjrqnfbzhsjv6zq2r0m48z-stdenv-bootstrap-tools --option binary-caches https://cache.nixos.org)/on-server/bootstrap-tools.tar.xz
nix store add-path --name busybox                 $(nix-store -r /nix/store/9y4qnwflnxkjrqnfbzhsjv6zq2r0m48z-stdenv-bootstrap-tools --option binary-caches https://cache.nixos.org)/on-server/busybox
@dguibert dguibert force-pushed the dg/new-bootstrap-tools-aarch64 branch from dd9010a to 8e7acfc Compare December 20, 2021 09:56
@dguibert
Copy link
Member Author

Ok. I've updated the PR with the native hydra job.

@tpwrules
Copy link
Contributor

I was able to use this PR to build a minimal system on an aarch64 machine with 16k pages. Before it was impossible to change the stdenv because the bootstrap tools would segfault. Thanks for fixing this.

Copy link
Member

@vcunat vcunat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I re-checked the URL and hashes, and also building some stuff (though it's only 4k-page machine).

@lovesegfault: we only need to upload this pair of files, again.

@dguibert
Copy link
Member Author

I re-checked the URL and hashes, and also building some stuff (though it's only 4k-page machine).

Great.
I knew about aarch64 build box for long time but I took time just now to ask access: NixOS/aarch64-build-box#122

@lovesegfault
Copy link
Member

Happy to help with uploading the new bootstrap tarball!

I need:

  1. The commit hash that was used to generate the bootstrap tools
  2. The store path of the bootstrap tools
  3. The sha256sum of all the on-server components, e.g.:
$ sha256sum /nix/store/9h4d7s313wv3gkfwi493yr1wvdsz9lf2-stdenv-bootstrap-tools/on-server/*
6fae5d5db226ea8eace94f2d0221a947a48c7ef7e7fd515c6604c705eb49662b  /nix/store/9h4d7s313wv3gkfwi493yr1wvdsz9lf2-stdenv-bootstrap-tools/on-server/bootstrap-tools.cpio.bz2
78df72595b11e55ab198a6508cd9fa6702baee33e9631d6f0614cf08052f88f3  /nix/store/9h4d7s313wv3gkfwi493yr1wvdsz9lf2-stdenv-bootstrap-tools/on-server/bzip2
467a81602423f7f8e73b59b5ae080235795aebbbcb19a4fb81c9706153615394  /nix/store/9h4d7s313wv3gkfwi493yr1wvdsz9lf2-stdenv-bootstrap-tools/on-server/cpio
6f9673b2e44bfa524068965da0ee5520f2c27f02f7dc3585ffe828356fb46eb1  /nix/store/9h4d7s313wv3gkfwi493yr1wvdsz9lf2-stdenv-bootstrap-tools/on-server/mkdir
865d209fe8fb0fb457983f0208bb5febe87990b800a378ecf5f1a1cd5574b65b  /nix/store/9h4d7s313wv3gkfwi493yr1wvdsz9lf2-stdenv-bootstrap-tools/on-server/sh

(I use this last bit to double check that the intended thing is being uploaded)

@dguibert
Copy link
Member Author

dguibert commented Jan 4, 2022

Happy to help with uploading the new bootstrap tarball!

Great.

I need:

1. The commit hash that was used to generate the bootstrap tools

c7c997a0662bf88264db52cbc41e67884eb7a1ff

2. The store path of the bootstrap tools

/nix/store/9y4qnwflnxkjrqnfbzhsjv6zq2r0m48z-stdenv-bootstrap-tools (from /nix/store/0azgpcnws27pva4m5spp1kfmmghf40gx-stdenv-bootstrap-tools/nix-support/hydra-build-products)

3. The `sha256sum` of all the `on-server` components, e.g.:
$ sha256sum /nix/store/9y4qnwflnxkjrqnfbzhsjv6zq2r0m48z-stdenv-bootstrap-tools/on-server/*
0233af99a5bc24555968149150c2ae7ebf64268ce0fedb19afb3ef504050ca2e  /nix/store/9y4qnwflnxkjrqnfbzhsjv6zq2r0m48z-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
9c9eba5f65dfb12b0ee2f97a34bcda159d4e521593c323e70e95dd6b4dbfe49d  /nix/store/9y4qnwflnxkjrqnfbzhsjv6zq2r0m48z-stdenv-bootstrap-tools/on-server/busybox

@lovesegfault
Copy link
Member

Done!

$ pushd ~/src/nixpkgs; git show c7c997a0662bf88264db52cbc41e67884eb7a1ff; popd                    
~/src/nixpkgs ~/tmp/bootstrap
commit c7c997a0662bf88264db52cbc41e67884eb7a1ff
Merge: 571cbf3d1db e2738be1365
Author: maralorn <[email protected]>
Date:   Fri Dec 3 01:27:14 2021 +0100

    Merge pull request #148357 from maralorn/nom-update
    
    nix-output-monitor: 1.0.3.3 -> 1.0.4.0

~/tmp/bootstrap

$ nix build /nix/store/9y4qnwflnxkjrqnfbzhsjv6zq2r0m48z-stdenv-bootstrap-tools                                 

$ sha256sum /nix/store/9y4qnwflnxkjrqnfbzhsjv6zq2r0m48z-stdenv-bootstrap-tools/on-server/*
0233af99a5bc24555968149150c2ae7ebf64268ce0fedb19afb3ef504050ca2e  /nix/store/9y4qnwflnxkjrqnfbzhsjv6zq2r0m48z-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
9c9eba5f65dfb12b0ee2f97a34bcda159d4e521593c323e70e95dd6b4dbfe49d  /nix/store/9y4qnwflnxkjrqnfbzhsjv6zq2r0m48z-stdenv-bootstrap-tools/on-server/busybox

$ aws s3 cp --recursive --acl public-read /nix/store/9y4qnwflnxkjrqnfbzhsjv6zq2r0m48z-stdenv-bootstrap-tools/on-server/ s3://nixpkgs-tarballs/stdenv-linux/aarch64/c7c997a0662bf88264db52cbc41e67884eb7a1ff                          
upload: ../../../../nix/store/9y4qnwflnxkjrqnfbzhsjv6zq2r0m48z-stdenv-bootstrap-tools/on-server/busybox to s3://nixpkgs-tarballs/stdenv-linux/aarch64/c7c997a0662bf88264db52cbc41e67884eb7a1ff/busybox
upload: ../../../../nix/store/9y4qnwflnxkjrqnfbzhsjv6zq2r0m48z-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz to s3://nixpkgs-tarballs/stdenv-linux/aarch64/c7c997a0662bf88264db52cbc41e67884eb7a1ff/bootstrap-tools.tar.xz

$ aws s3 ls s3://nixpkgs-tarballs/stdenv-linux/aarch64/c7c997a0662bf88264db52cbc41e67884eb7a1ff/
2022-01-04 11:27:11   16156444 bootstrap-tools.tar.xz
2022-01-04 11:27:11     177608 busybox

$ aws s3 cp --recursive s3://nixpkgs-tarballs/stdenv-linux/aarch64/c7c997a0662bf88264db52cbc41e67884eb7a1ff/ ./                                                                                            
download: s3://nixpkgs-tarballs/stdenv-linux/aarch64/c7c997a0662bf88264db52cbc41e67884eb7a1ff/busybox to ./busybox
download: s3://nixpkgs-tarballs/stdenv-linux/aarch64/c7c997a0662bf88264db52cbc41e67884eb7a1ff/bootstrap-tools.tar.xz to ./bootstrap-tools.tar.xz

$ sha256sum ./bootstrap-tools.tar.xz ./busybox                                            
0233af99a5bc24555968149150c2ae7ebf64268ce0fedb19afb3ef504050ca2e  ./bootstrap-tools.tar.xz
9c9eba5f65dfb12b0ee2f97a34bcda159d4e521593c323e70e95dd6b4dbfe49d  ./busybox

@vcunat vcunat merged commit 0b7de2a into NixOS:staging Jan 4, 2022
@vcunat
Copy link
Member

vcunat commented Jan 4, 2022

I tried to build some stuff. No use waiting anymore.

@tpwrules
Copy link
Contributor

tpwrules commented Jan 4, 2022

I cleaned up my 16k page aarch64 machine and was able to run nixpkgs-review pr 151399 -p hello. Looks like everything was uploaded correctly and the full stdenv bootstrap works (along with hello of course).

Looking forward to seeing this land in master! Thanks everyone for tackling this.

@dguibert dguibert deleted the dg/new-bootstrap-tools-aarch64 branch January 4, 2022 21:51
@dguibert dguibert restored the dg/new-bootstrap-tools-aarch64 branch January 4, 2022 21:51
@dguibert dguibert deleted the dg/new-bootstrap-tools-aarch64 branch January 4, 2022 21:51
@dguibert dguibert restored the dg/new-bootstrap-tools-aarch64 branch January 12, 2022 09:44
@ghost ghost mentioned this pull request Jul 5, 2022
3 tasks
fgaz added a commit to fgaz/nixpkgs that referenced this pull request Jan 21, 2024
This PR updates the bootstrap tarballs for riscv64-linux with new Hydra-generated ones.

Fixes NixOS#275848 (bootstrap assembler too old to build gcc 13).

I'll be following the script used in NixOS#151399, NixOS#168199, NixOS#183487, and NixOS#188334.

Files came from [this](https://hydra.nixos.org/build/246376732#tabs-summary) Hydra build, which used nixpkgs revision 160cedc to instantiate:

```
/nix/store/cpiajh4l83b08pynwiwkpxj53d78pcxr-stdenv-bootstrap-tools-riscv64-unknown-linux-gnu.drv
```

and then built:

```
/nix/store/8a92pj40awdw585mcb9dvm4nyb03k3q3-stdenv-bootstrap-tools-riscv64-unknown-linux-gnu
```

I downloaded these files from Hydra and prefetched them into the nix store with the following commands:

```
STOREPATH=8a92pj40awdw585mcb9dvm4nyb03k3q3-stdenv-bootstrap-tools-riscv64-unknown-linux-gnu
OPTIONS="--option binary-caches https://cache.nixos.org --option trusted-public-keys cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
nix store prefetch-file \
  file://$(nix store add-file --name bootstrap-tools.tar.xz  $(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/bootstrap-tools.tar.xz)
nix store prefetch-file --executable \
  file://$(nix store add-path --name busybox $(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/busybox)
```

These commands produced the following output:

```
Downloaded 'file:///nix/store/xs74wcxq3qc12plfc70fds8inbndhcfm-bootstrap-tools.tar.xz' to '/nix/store/3fal4gikp92013kac6rdmfbrch2s859b-xs74wcxq3qc12plfc70fds8inbndhcfm-bootstrap-tools.tar.xz' (hash 'sha256-0LxRd7fdafQezNJ+N2tuOfm0KEwgfRSts5fhP0e0r0s=').
Downloaded 'file:///nix/store/9ndpna6jrlac4y9fappdjm0sxx0g2bja-busybox' to '/nix/store/kb7wyy30y1gxcmdajljr26kxxac606qa-9ndpna6jrlac4y9fappdjm0sxx0g2bja-busybox' (hash 'sha256-OGO96QUzs2n5pGipn/V87AxzUY9OWKZl417nE8HdZIE=').
```

I used the hashes from the output above to create the `fetchurl` invocation which is part of this commit.

I then started the bootstrap with the following command:

```
nix build -L -f . --arg localSystem '(import ./lib).systems.examples.riscv64' hello
```

As @lovesegfault requested, here are the the `sha256sum`s of all the `on-server` components for extra verification:

```
sha256sum /nix/store/${STOREPATH}/on-server/*
```

which produced the following output:

```
d0bc5177b7dd69f41eccd27e376b6e39f9b4284c207d14adb397e13f47b4af4b  /nix/store/8a92pj40awdw585mcb9dvm4nyb03k3q3-stdenv-bootstrap-tools-riscv64-unknown-linux-gnu/on-server/bootstrap-tools.tar.xz
65f9433abb598f63c932d33351b14f686551512b1cece1e64c2d0e76aa0ec52e  /nix/store/8a92pj40awdw585mcb9dvm4nyb03k3q3-stdenv-bootstrap-tools-riscv64-unknown-linux-gnu/on-server/busybox
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
6.topic: stdenv Standard environment 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants