Evaluation error is blocking jobsets

[FRidh]

Describe the bug

hydra-eval-jobs returned exit code 1:
warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy
warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy
warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy
warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy
warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy
trace: `mkStrict' is obsolete; use `mkOverride 0' instead.
trace: `lib.nixpkgsVersion` is deprecated, use `lib.version` instead!
trace: Warning: `showVal` is deprecated and will be removed in the next release, please use `traceSeqN`
trace: lib.zip is deprecated, use lib.zipAttrsWith instead
warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy
warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy
skipping job with illegal name 'http_parser.rb'
skipping job with illegal name 'http_parser.rb'
skipping job with illegal name 'http_parser.rb'
skipping job with illegal name 'http_parser.rb'
skipping job with illegal name 'sourcemap.vim'
trace: `mkStrict' is obsolete; use `mkOverride 0' instead.
trace: `lib.nixpkgsVersion` is deprecated, use `lib.version` instead!
trace: Warning: `showVal` is deprecated and will be removed in the next release, please use `traceSeqN`
trace: lib.zip is deprecated, use lib.zipAttrsWith instead
warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy
warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy
warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy
warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy
warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy
warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy
warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy
error: unexpected EOF reading a line

The actual issue is masked by a Hydra bug NixOS/hydra#728.

[knedlsepp]

There's more instances of the evaluation error even:
https://hydra.nixos.org/jobset/nixos/release-20.03#tabs-errors
https://hydra.nixos.org/jobset/nixos/trunk-combined#tabs-errors
...

[vcunat]

I'm unable to reproduce any error locally (without hydra). I did an experiment on unstable-small jobset: the last successful evaluation is on 05f0934, so I forced it to re-try with the same commit and it failed with

hydra-eval-jobs returned exit code 1:
error: unexpected EOF reading a line

On the other hand, 19.09 seems unaffected 🤷‍♂️ this seems just hard when the real errors get hidden.

[samueldr]

Likely fix for the Hydra issue:

• hydra-eval-jobs: Mirror eval errors in STDERR hydra#729

Once fixed, evals should show errors again.

[samueldr]

It may be a nix-side regression!

While my nix-daemon was /nix/store/nk4px4bjp0kiss27n5dyrwsj9xgflwhp-nix-2.3, it successfully eval'd nixos/release-small.nix.

Switching to the revision of nix shown on the Hydra footer (3e7aab8) the issue is cleanly reproducible.

I'll add that it also reproduces quickly! 1m10.799s, rather than more than an hour!

[samueldr]

In Nix:

There are only 'skip'ped commits left to test.
The first bad commit could be any of:
401b5bc5418f3eb6d57da9d9e66df055f8bce122
75db069f927ffaf38ac6ef2d8143926b724ca935
d700eecea9a274c1b45549141f40180ac74454ce
22a754c091f765061f59bef5ce091268493bb138
887030f211dcd062a73021b1cc289992502b35e4
d37dc71e3cf077fa5d24a9bf8395deae21cc4410
We cannot bisect more!

• NixOS/nix@401b5bc
• NixOS/nix@75db069
• NixOS/nix@d700eec (only adds a test)
• NixOS/nix@22a754c
• NixOS/nix@887030f (unrelated emacs thing)
• NixOS/nix@d37dc71

All had to be skipped as the tests won't pass.

---

EDIT: It looks like NixOS/nix@75db069 may be the regression.

[edolstra]

Yes, it's a known issue: NixOS/nix#3462

[samueldr]

See also: #83863

[flokli]

Just to make sure, is NixOS/nix#3462 currently still blocking 20.03 evaluations? Can we move hydra.nixos.org to a version before that?

I think having done some Hydra evaluations is critical to do before the 20.03 release.

cc @disassembler @worldofpeace

[jtojnar]

We reverted #73966 so this particular instance that triggered NixOS/nix#3462 is no longer happening and the evals are running: https://hydra.nixos.org/job/nixos/release-20.03/tested#tabs-status

[samueldr]

This was fixed by @jtojnar, closing as per the short discussion in the go-no-go meeting

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/go-no-go-meeting-nixos-20-03-markhor/6495/16

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixos-unstable-hasnt-been-updated-in-a-while/6584/8

[andir]

It appears like this issue has returned to the unstable channel. All that is being logged is a JSON error (something is null).

[vcunat]

It will surely come from a different place, but we can recycle this issue.

[samueldr]

If this is like #99236, I have doubts that this is related.

The following error, for #99236, didn't result in an error message being eaten by the hydra evaluator.

error: [json.exception.type_error.302] type must be string, but is null

While in the past the following would:

error: unexpected EOF reading a line

It would be better to instead follow-up in a new issue (with linking to suspect previous issues) imo.

---

What's that with error messages being eaten by the hydra evaluator?

• hydra-eval-jobs masking real errors hydra#728

And a new instance found (unrelated to the current issue)

• hydra-eval-jobs masks error messages from Nix (again) hydra#822

[andir]

The eval issue is gone. It turned out to be due to one of the changes I made to the test infrastructure. This has been fixed in #101645.

As a result of the joined work on this @samueldr did propose NixOS/hydra#825 which would (hopefully) allow us to spot the source of these errors much quicker. Usually the eval output on hydra is very noisy and any pointer towards where the actual error is coming from is helpful.