Document how to test packages automatically

[davidak]

We have documentation on how to write NixOS (module) tests.

https://github.com/tfc/nixpkgs/blob/40396a7e12419fc772063d2a3fa87ec7d14a46d9/nixos/doc/manual/development/writing-nixos-tests.xml

But how to write tests for packages?

I asked on IRC and got some helpful informations:

<davidak> are the nixos tests also intended for applications or only for nixos modules? there is for example chromium and fish, but mostly modules
<davidak> i want to add tests for my cli application packages
<gchristensen> they have a very high cost to run
<clever> davidak: for cli apps, its usually simpler to just put it into the derivation, as a checkPhase
<clever> i think the only reason things like chromium are in a nixos test, is because it needs X11 to test
<clever> so chromium itself isnt a module, but it relies on a module to start
...
<davidak> clever, this is a nice simple example of a test every program should have https://github.com/NixOS/nixpkgs/blob/076860e0340a5e4a909b9a710e186508b14d1c90/pkgs/development/tools/scalafix/default.nix#L34

<davidak> clever, eny idea how to test programs that connect to the intenet like `tldr --update`
<clever> davidak: the internet is disabled during all testing
<davidak> clever, so the answer ist "we can't test such cases"? that would be bad :/
gchristensen> we can't actually
<clever> davidak: what exactly is it fetching from the internet? could you fake it with an nginx server and /etc/hosts?
<davidak> i think it clones a git repo or download some archive
* mananamenos has quit (Ping timeout: 240 seconds)
<davidak> with that method it would need a nixos module test, but i think that's too costly for such a small program just a few people use

So my goal with this issue is to get this information documented.

• explain the difference between NixOS module tests and package tests
• explain when to use which
• link to good examples for both
• create one example for a module test with nginx to test a package that needs internet access to download stuff

We maybe have to decide on a policy when a test should be added and when not (e.g. it's not important enough to use costly module tests)

Related issues:

• Add installed tests #34987
• Package tests #27604
• Test all nix expressions where applicable #46632
• [RFC] nixpkgs/tests: create nixos-independent PoC tests #36842

[davidak]

There is also passthru.tests to link to test derivations from a package.

https://nixos.org/nixpkgs/manual/#sec-standard-meta-attributes

It should be called meta.tests initially, but it's not possible due to technical limitations.

NixOS/nix#2532 (comment)
#44439

[davidak]

@jtojnar commented in #27604 (comment)

@davidak IMHO checkPhase is not a good place since it cannot be run independently of the package build. Tweaking the test would mean rebuilding the whole package, and so would disabling flaky test.

Me and @worldofpeace are adding new nixos tests to passthru.tests (example). r-ryantm runs derivations listed in that attribute (nix-community/nixpkgs-update#118) and ofborg might do that as well (NixOS/ofborg#410).

I think using runCommand in there might also work nicely.

So this looks like the way to go for now!

[stale]

Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the
   related code and @ mention them in a comment.
2. Ask on the NixOS Discourse. 3. Ask on the #nixos channel on
   irc.freenode.net.

[davidak]

i still want to do it

waiting for markdown docs

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-many-people-are-paid-to-work-on-nix-nixpkgs/8307/51

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/contributor-retention/12412/45