Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🦝 NixOS 22.11 - Zero Hydra Failures #199919

Closed
mweinelt opened this issue Nov 6, 2022 · 12 comments
Closed

🦝 NixOS 22.11 - Zero Hydra Failures #199919

mweinelt opened this issue Nov 6, 2022 · 12 comments
Labels
0.kind: ZHF Fixes Fixes during the Zero Hydra Failures (ZHF) campaign

Comments

@mweinelt
Copy link
Member

mweinelt commented Nov 6, 2022

Hi, we are Janne Heß & Martin Weinelt, the release managers for NixOS 22.11 🦝 ("Raccoon").

Today we want to invite everyone to participate in the Zero Hydra Failures Project, wherein we prepare the package set for the upcoming release, up until its public release at the end of november.

There are only two more upcoming dates that we'd like to mention in that context:

  • 2022-11-21: Branch-off
  • 2022-11-30: 22.11 Release

The complete timeline can be found in the

The mission

Every time we plan to do a release, we take time to stabilize the master branch and later on the release branch. Our goal here is to reduce the number of failing jobs on the nixpkgs:trunk nixos:trunk-combined jobsets as much as possible before branch-off. We call this the "Zero Hydra Failure" campaign.

Besides aiming for zero failed jobs, we also strive to again provide all packages that were available in the previous release.

Changes need to always target the master branch. Take note that the branch-off will occur on 2022-11-21, after which ZHF changes will need to be tagged with the backport: release-22.11 label to land in the stable release.

Jobsets

The relevant jobsets to check for failing jobs are:

Workflow

Finding broken packages

Eval reports

Evaluation reports provide a structural overview of the most impactful failing builds. They originated at https://github.com/nix-community/nix-review-tools and were automated over at https://github.com/malob/nix-review-tools-reports.

  1. Navigate to https://malob.github.io/nix-review-tools-reports/
  2. Open the relevant jobset
  3. Browse the latest reports for build failures
  4. Follow the links to the build failure on hydra

ZERO Hydra Failures

The platform automatically crawls Hydra and lists packages by maintainer and lists the most important dependencies (failing packages with the most dependants). It also graphs the general trend per platform.

  1. Navigate to https://zh.fail

For the record, we started ZHF here:

Latest Linux evaluation (completely built):	1784776 on 2022-11-05 07:52:49 (UTC)
Latest Darwin evaluation (completely built):	1784782 on 2022-11-05 09:24:59 (UTC)
Failing builds on aarch64-linux:	1706
Failing builds on i686-linux:	262
Failing builds on x86_64-darwin:	1057
Failing builds on x86_64-linux:	2695
Total failed builds	5720

Check on packages you maintain

  1. Clone nixpkgs and checkout the master branch
  2. Run
    nix-build maintainers/scripts/build.nix --argstr maintainer <name>
    

Hydra

Hydra is nixpkgs CI platform, where all active branches are built and pushed into the cache, after which channels can originate from its build results.

  1. Open the nixpkgs:trunk jobset
  2. Select the latest evaluation
  3. Directly failing jobs are marked with a red cross, while transitively failing ones are greyed out.
  4. Use the search form to scope the package list to things relevant to you and that you can test.

Submit fixes

  1. Search through PRs to make sure noone provided a fix yet. If there is one, please take the time and help review the change.

  2. If there is no open PR, troubleshoot why it's failing and fix it.

  3. Pull Request the fix against the master branch and wait potential review & change requests

    • If your PR causes more than ~500 rebuilds, it is generally preferred to target staging to avoid compute churn for users on master.

Backporting

After 2022-11-21

  1. Apply the relevant backport label to land the fix in the release branch

    • Changes to master get backported into release-22.11
    • Changes to staging get backported into staging-22.11
  2. If the backport action fails, follow the manual backporting steps. Make sure to use git cherry-pick -x <rev> on all commits intended for backport.


Always link back to this issue by mentioning the issue number in the description of your pull request:

ZHF: #199919

If your PR receives no reviews or does not get merged, feel free to

  • add the 0.kind: ZHF Fixes label, so people can better browse these fixes
  • request or mention @NixOS/nixos-release-managers on the PR

Broken packages

Everything we cannot fix in time will need to be marked broken on the respective platforms, so that Hydra will not retry builds over and over, thereby wasting compute resources.

Set meta.broken and add a reference and/or explanation, like this:

meta = {
  # ref to issue/explanation
  broken = stdenv.isDarwin; # only broken on darwin
  # broken = true; # broken on all platforms.
};

Closing

This is a great way to help NixOS, and it is a great time for new contributors to start their nixpkgs adventure. 🥳

As with the feature freeze issue, please keep the discussion here to a minimum so we don't ping all maintainers (although relevant comments can of course be added here if they are directly ZHF-related) and ping one of the release managers (@mweinelt, @dasJ) in the respective issues.

@mweinelt
Copy link
Member Author

mweinelt commented Nov 6, 2022

Reserved post.

cc @NixOS/nixpkgs-committers @NixOS/nixpkgs-maintainers

@StarGate01 StarGate01 reopened this Nov 6, 2022
@nixos-discourse
Copy link

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/zero-hydra-failures-22-11-edition/23054/1

@mweinelt mweinelt added the 0.kind: ZHF Fixes Fixes during the Zero Hydra Failures (ZHF) campaign label Nov 6, 2022
@amaxine amaxine mentioned this issue Nov 6, 2022
13 tasks
@mweinelt mweinelt pinned this issue Nov 6, 2022
@raphaelr raphaelr mentioned this issue Nov 6, 2022
13 tasks
reckenrode added a commit to reckenrode/nixpkgs that referenced this issue Nov 6, 2022
reckenrode added a commit to reckenrode/nixpkgs that referenced this issue Nov 7, 2022
reckenrode added a commit to reckenrode/nixpkgs that referenced this issue Nov 7, 2022
@viraptor viraptor mentioned this issue Nov 7, 2022
13 tasks
thenonameguy added a commit to thenonameguy/nixpkgs that referenced this issue Nov 29, 2022
Bump version to fix build failure for ZHF (NixOS#199919)
@mweinelt mweinelt closed this as completed Dec 1, 2022
@mweinelt mweinelt unpinned this issue Dec 1, 2022
rtimush pushed a commit to rtimush/nixpkgs that referenced this issue Sep 21, 2023
Fixes build failure due to missing depend:

    *** No intltoolize found, please install the intltool package ***

ZHF: NixOS#199919
rtimush pushed a commit to rtimush/nixpkgs that referenced this issue Sep 21, 2023
Update UHDM and also fix for Darwin build.
The gtest version used in the project unconditionally uses
/tmp as temporary directory in Darwin and does not honor the
environment variables introduced in gtest 1.12.
Use that gtest version so that it will work in the Hydra
build environment.

Related: NixOS#199919 as this is fixing the Darwin build.

Signed-off-by: Henner Zeller <[email protected]>
rtimush pushed a commit to rtimush/nixpkgs that referenced this issue Sep 21, 2023
Update Surelog (going along with uhdm 1.45). Fix flaky tests
to to parallel execution of ctest by patching in a fix that is
already upstream.

Result zero hydra failures NixOS#199919

Also: using flatbuffers from nix environment instead of third_party/

Signed-off-by: Henner Zeller <[email protected]>
rtimush pushed a commit to rtimush/nixpkgs that referenced this issue Sep 21, 2023
Without the change build fails as:

    ...-linux-headers-6.0/include/linux/fiemap.h:37:30:
      error: flexible array member 'fiemap::fm_extents' not at end of 'struct crucible::Fiemap'
       37 |         struct fiemap_extent fm_extents[]; /* array of mapped extents (out) */
          |                              ^~~~~~~~~~

Upstream ported to the new version but did not release compatible
version yet. As patch backport is a bit convoluted let's wait
for a new release and pin to previous linuxHeaders_5_19 version.

ZHF: NixOS#199919
rtimush pushed a commit to rtimush/nixpkgs that referenced this issue Sep 21, 2023
Without the change build fails on hydra due to `gcc` version mix
between `opencolorio` and `djv`:

    [ 51%] Linking CXX executable ../../build/bin/Render2DStressTest
    ...-binutils-2.39/bin/ld: ...-opencolorio-1.1.1/lib/libOpenColorIO.so:
      undefined reference to `std::__throw_bad_array_new_length()@GLIBCXX_3.4.29'

ZHF: NixOS#199919
rtimush pushed a commit to rtimush/nixpkgs that referenced this issue Sep 21, 2023
Currently broken, should be fixed a bit later
see last comments in NixOS#200573

ZHF: NixOS#199919
rtimush pushed a commit to rtimush/nixpkgs that referenced this issue Sep 21, 2023
rtimush pushed a commit to rtimush/nixpkgs that referenced this issue Sep 21, 2023
Was able to build this package after updating hashes.
Hydra failures don't mention hash mismatch, but maybe update will still help.

ZHF: NixOS#199919
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0.kind: ZHF Fixes Fixes during the Zero Hydra Failures (ZHF) campaign
Projects
None yet
Development

No branches or pull requests

6 participants