Add installed tests

[jtojnar]

We want to test that packages work on NixOS but writing tests manually is a chore so not many applications are covered. Fortunately, many packages (especially from GNOME or Freedesktop) ship so called installed tests that already do what we want.

I already implemented them for gjs and fwupd and as you can see it is actually quite simple. We can even factor it out into something like:

import ./make-test.nix ({ pkgs, ... }:
  let
    makeInstalledTest = { tested, withX ? false }: {
      name = tested.name;

      meta = {
        maintainers = tested.meta.maintainers;
      };

      machine = { pkgs, ... }: {
        imports = stdenv.lib.optional withX ./common/x11.nix;
        environment.systemPackages = with pkgs; [ gnome-desktop-testing ];
        environment.variables.XDG_DATA_DIRS = [ "${tested.installedTests}/share" ];
      };

      testScript = ''
        ${stdenv.lib.optional withX "$machine->waitForX;"}
        $machine->succeed("gnome-desktop-testing-runner");
      '';
    };
  in makeInstalledTest { tested = pkgs.gjs; withX = true; }
)

Here is incomplete list of software that provides installed tests:

• clutter
• cogl
• colord
• eog
• evolution
• evolution-data-server
• folks
• fwupd
• flatpak
• flatpak-builder (flatpak-builder: 1.0.6 -> 1.0.7 #62413)
• gdk-pixbuf
• geocode-glib
• gjs
• glib
• glib-networking
• glib-openssl
• gnome-boxes
• gnome-calculator
• gnome-characters
• gnome-credentials
• gnome-desktop
• gnome-ostree
• gnome-photos
• gnome-software
• gnome-weather
• gspell
• gtk+
• gtksourceview
• gvfs
• ibus-engines.hangul
• json-glib
• libgdata
• libgrss
• libgsystem
• libmediaart
• librsvg
• libsoup
• mutter
• ostree
• pango
• tepl
• tracker
• tracker-miners
• xdg-desktop-portal

Finally, we should connect the tests to their respective packages, for example using #27604

[jtojnar]

Ibus apparently has something based on gdtr fujiwarat/ibus-anthy@5a9e485, we should probably use that to prevent things like #68837 from reoccurring.

[stale]

Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the
   related code and @ mention them in a comment.
2. Ask on the NixOS Discourse. 3. Ask on the #nixos channel on
   irc.freenode.net.

[jtojnar]

This is a long term tracking goal.

[stale]

I marked this as stale due to inactivity. → More info

[willbush]

Hello! I'd like to start contributing to nixpkgs. I was thinking to try to improve testing to prevent regressions. Is this a good issue to start with?

[jtojnar]

@willbush Hi. It will depend on your experience and vary from package to package. For libraries that are mostly self-contained, it may be as simple as adding the tested package and making sure the installed test files are installed by the package (usually there will be an configure option, though sometimes it is broken because upstreams do not regularly run installed tests). Sometimes there are paths that need to be patched or extra dependencies to be added, just like with any other Nix packaging. With extra dependencies, it might be nice to move the installed tests into a separate output.

Here are recent commits introducing installed tests to geocode-glib as another example: fb2877c 6a7da5b

json-glib, pango, libsoup, gtksourceview, evolution-data-server, evolution, tracker, gvfs, tracker-miners are the more consequential packages where tests might be useful (ordered roughly by the estimated complexity of getting the tests running, easiest first).

Feel free to ping me on Matrix or Discourse if you have more specific questions.

[willbush]

@jtojnar Took a while to get over the learning curve, but I think json-glib PR is inline with how others were done.

I noticed that graphene and json-glib tests execute very fast. I was wondering why a VM has to be spun up for them? Is it because we don't know how they modify their environment?

[willbush]

@jtojnar I noticed that gtksourceview5 runs meson test in the checkPase. Would installed tests still be desired?

[jtojnar]

@willbush Sorry, this got lost in the noise.

I noticed that graphene and json-glib tests execute very fast. I was wondering why a VM has to be spun up for them? Is it because we don't know how they modify their environment?

I think for installed tests it is desirable to make them closer to a real runtime environment.

I noticed that gtksourceview5 runs meson test in the checkPase. Would installed tests still be desired?

Looks like the installed tests of gtksourceview5 run the same set of tests as the test suite so installing them is probably not necessary.

I would say that the installed tests mainly make sense for tests that require realistic runtime environment or ones that take long time. If the test suite is mostly self-contained, running it in the build is sufficient.