nixos-unstable evaluation broken on Hydra

[vcunat]

and thus blocking nixos-unstable channel (for a few days already). I have no idea what's the problem yet: https://hydra.nixos.org/jobset/nixos/trunk-combined#tabs-errors

[vcunat]

I don't know... I'm giving up now, at the very least for today.

[jonringer]

the xorg fonts were addressed in #99136

[andersk]

I’ve been trying to figure out what happened here.

I can reproduce the evaluation problem in about an hour by running

nix-shell -p hydra-unstable --run 'hydra-eval-jobs -I . nixos/release-combined.nix --verbose'

Using git bisect, I traced the problem to c451603, for which the output ends with

evaluating file '/home/anders/nixpkgs/pkgs/development/libraries/audio/ntk/default.nix'
evaluating file '/home/anders/nixpkgs/pkgs/development/compilers/zz/default.nix'
evaluating file '/home/anders/nixpkgs/pkgs/tools/security/zzuf/default.nix'
evaluating file '/home/anders/nixpkgs/pkgs/build-support/release/default.nix'
evaluating file '/home/anders/nixpkgs/nixos/release.nix'
evaluating file '/home/anders/nixpkgs/nixos/lib/make-channel.nix'
evaluating file '/home/anders/nixpkgs/pkgs/build-support/release/source-tarball.nix'
error: [json.exception.type_error.302] type must be string, but is null

However, as you’ve seen, reverting c451603 doesn’t fix the evaluation; there’s a second problem. Using a second git bisect with c451603 reverted at every step, I traced the second problem to fb6d63f, for which the output ends with

copying path '/nix/store/vk79i7s053v4iy3nlps502g4phxszjnp-binutils-wrapper-2.31.1' from 'https://cache.nixos.org'...
downloading 'https://cache.nixos.org/nar/0bfl4yzdrsw62zyrmjsq9lmp9zw2ff9lz701fy7j5bazrzj3aslx.nar.xz'...
copying path '/nix/store/pjimnayw2iyrz7m6rf09w47lrc03dx8q-gdk-pixbuf-2.40.0' from 'https://cache.nixos.org'...
downloading 'https://cache.nixos.org/nar/0ksv8apw7wy5m0aa70yl6ci901ca1kswqim8wwn4vnx6nkgyawxn.nar.xz'...
copying path '/nix/store/5ylqw9z9qswf4hvs1lldgj4cyjm348kf-libnotify-0.7.9' from 'https://cache.nixos.org'...
downloading 'https://cache.nixos.org/nar/0isg1hkfqc4221nclgqmy2m6xprfy42ldvwn7yn04hagi3lfmpmx.nar.xz'...
copying path '/nix/store/4hwcjapw28w2w8gqar5dxmk9vpglp6iz-gcc-wrapper-9.3.0' from 'https://cache.nixos.org'...
downloading 'https://cache.nixos.org/nar/072cz826qg2c9xh9kkf1jqy9idscbn6syg2hrizb4h7zayrb2yn7.nar.xz'...
copying path '/nix/store/0g0aiwhrcz8a6c4cr32kz428qsgrpbp2-stdenv-linux' from 'https://cache.nixos.org'...
downloading 'https://cache.nixos.org/nar/1lh6dqk9952gih663bn2hch5rb5q1nsy3b4g9hxz48in1bbaq45p.nar.xz'...
error: "\u001b[31;1merror:\u001b[0m\u001b[34;1m --- Error ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- hydra-eval-jobs\u001b[0m\na 'aarch64-linux' with features {} is required to build '/nix/store/dhwnmrcdsz42d7sb9j2hb19z4j66wnql-apparmor-parser-2.13.4.drv', but I am a 'x86_64-linux' with features {benchmark, big-parallel, kvm, nixos-test}"
error: --- EndOfFile ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ hydra-eval-jobs
unexpected EOF reading a line

In both instances, these error messages and failure modes are much less clear than they should be…

[vcunat]

I suspect Hydra.nixos.org skips over that due to different settings; in its output I see:

error: "error: --- EvalError --- hydra-eval-jobs\nattempted to realize '/nix/store/4shyj8ckj7f6lxy5jvk0d9spx99dhgsi-apparmor-utils-2.13.4.drv' during evaluation but 'allow-import-from-derivation' is false"

(multiple times and lots of other things under so clearly it doesn't stop there)

[vcunat]

To be clear, build-during-eval in there is certainly bad, but I'm not convinced it's a blocker here.

[andersk]

It’s certainly not impossible that hydra.nixos.org has different settings. But I would note that this vanilla hydra-eval-jobs command does succeed on current nixos-unstable and on the parent commits of the commits I noted. If my (lack of) settings are what caused it to fail, it would be quite a coincidence for it to have suddenly started to fail now.

[eyJhb]

Is there a new issue for tracking the progress/findings/state?

[vcunat]

This is the issue. I'm hopeful that the jobset will evaluate now. (First attempt failed but that might be a random error; the message is different than some of the earlier ones.)

[eyJhb]

Sorry, didn't seem clear to me from context, that it works that way :)

[vcunat]

I confirm the means of local testing (though it's rather poor): on recent master before the revert (41c0f49) I ran:

nix-shell -p hydra-unstable --run 'hydra-eval-jobs -I . nixos/release-combined.nix --option allow-import-from-derivation false --verbose'

which ended in

evaluating file '/home/admin/nix/nixpkgs-x/pkgs/shells/zsh/zsh-powerlevel9k/default.nix'
evaluating file '/home/admin/nix/nixpkgs-x/pkgs/shells/zsh/zsh-prezto/default.nix'
evaluating file '/home/admin/nix/nixpkgs-x/pkgs/shells/zsh/zsh-syntax-highlighting/default.nix'
evaluating file '/home/admin/nix/nixpkgs-x/pkgs/shells/zsh/zsh-you-should-use/default.nix'
evaluating file '/home/admin/nix/nixpkgs-x/pkgs/misc/emulators/zsnes/default.nix'
evaluating file '/home/admin/nix/nixpkgs-x/pkgs/tools/networking/zssh/default.nix'
evaluating file '/home/admin/nix/nixpkgs-x/pkgs/tools/compression/zsync/default.nix'
evaluating file '/home/admin/nix/nixpkgs-x/pkgs/data/themes/zuki/default.nix'
evaluating file '/home/admin/nix/nixpkgs-x/pkgs/applications/networking/instant-messengers/zulip/default.nix'
evaluating file '/home/admin/nix/nixpkgs-x/pkgs/applications/networking/instant-messengers/zulip-term/default.nix'
evaluating file '/home/admin/nix/nixpkgs-x/pkgs/development/python-modules/zulip/default.nix'
evaluating file '/home/admin/nix/nixpkgs-x/pkgs/development/python-modules/matrix-client/default.nix'
evaluating file '/home/admin/nix/nixpkgs-x/pkgs/development/python-modules/urwid-readline/default.nix'
evaluating file '/home/admin/nix/nixpkgs-x/pkgs/development/compilers/zulu/default.nix'
evaluating file '/home/admin/nix/nixpkgs-x/pkgs/development/compilers/zulu/8.nix'
evaluating file '/home/admin/nix/nixpkgs-x/pkgs/development/libraries/zxcvbn-c/default.nix'
evaluating file '/home/admin/nix/nixpkgs-x/pkgs/tools/graphics/zxing/default.nix'
evaluating file '/home/admin/nix/nixpkgs-x/pkgs/applications/audio/zynaddsubfx/default.nix'
evaluating file '/home/admin/nix/nixpkgs-x/pkgs/applications/audio/lash/default.nix'
evaluating file '/home/admin/nix/nixpkgs-x/pkgs/development/libraries/audio/ntk/default.nix'
evaluating file '/home/admin/nix/nixpkgs-x/pkgs/development/compilers/zz/default.nix'
evaluating file '/home/admin/nix/nixpkgs-x/pkgs/tools/security/zzuf/default.nix'
evaluating file '/home/admin/nix/nixpkgs-x/pkgs/build-support/release/default.nix'
evaluating file '/home/admin/nix/nixpkgs-x/nixos/release.nix'
evaluating file '/home/admin/nix/nixpkgs-x/nixos/lib/make-channel.nix'
evaluating file '/home/admin/nix/nixpkgs-x/pkgs/build-support/release/source-tarball.nix'
error: [json.exception.type_error.302] type must be string, but is null

(In my case the single evaluation takes around 110 minutes. It might be due to rotating HDD slowing down the sqlite transactions, though atop didn't report the drive as busy for significant fraction of the time.)

[vcunat]

Evaluation now really did succeed on the main jobset, so I think that mostly concludes this issue:

• Thanks Anders a lot!
• We want the channel to update, but if issues are encountered they'll probably be of a different nature and thus better suited for a separate thread.
• Maybe some more analysis etc. will come in this thread.

[andersk]

Awesome! My first two questions are definitely 1. what the fried cucumber does error: [json.exception.type_error.302] type must be string, but is null actually mean, and 2. can we fix this error to say what it actually means?

[vcunat]

I think this exception mainly needs a trace. (but I have no insight into implementation of this)

[eyJhb]

@andersk I actually got that error before, at some point long ago. But didn't make a issue for it, because I was just told I was doing something wrong.

Had a minimal example back then, but.. It got lost (maybe some place in the logs) - but it is like @vcunat said, needs a trace.

[7c6f434c]

So is hydra-eval-jobs enough to check if something weird happens with a fixed version of apparmor updates?

[vcunat]

Sounds enough to me. (when it includes --option allow-import-from-derivation false, of course) The main downside I know is that it takes lots of time and resources.