Require copr build job definition for running tests

[lbarcziova]

Currently, users can define only the test job and we will run Copr build before triggering the tests (if skip_build=False). With adding new features (identifiers, monorepo support, Image builder, ..) this has been causing a lot of "guessing" and updating code to match both the explicit definition of the build job and implicit.

We discussed we should start requiring a Copr build job definition for running tests:

• open PRs with updated Packit configurations of projects where the build job definition is missing
  • this research can be used
• update the documentation
• (have some "deprecation" period: in this period we could give e.g. a comment to PRs/commits if the Packit configuration misses build job and warn users about upcoming change) - maybe not needed if we do the updating of configs
• after that, if there will be a test job defined (skip_build=False) without any build job, we will report to user that the build definition is required via statuses and not run the build+tests

[FrNecas]

Since we have to introduce a mechanism for the config updates, there are some issues that need to be further discussed, mainly concerning rate limiting. I am putting my notes here (in case I don't manage to finish the card and someone else started working on this).

We want to store the packit configs (the current state on the main/master branch) in the database:

• introduce a handler that reacts to git push, this is simple. Can we somehow detect the default branch or do we have to hardcode master/main and not support other possible names?
• to start with, we do not want to wait for projects to have a push/merge to main, we want to know the current state of configs without any user action so that we can find projects that need modifications.

This means that we have to fetch the configs. This is problematic due to rate limiting:

• on prod there are ~33,000 git projects enabled, for each we have to check 4 possible packit names using /repos/{owner}/{repo}/contents/{path} Github endpoint (or Gitlab equivalent). This requires at least 120,000 requests.
• Github organization token seems to have a limit of 1000 requests per hour, personal token 5000 per hour according to the docs.
• It does not seem feasible to try to fetch the configs during database migration (this would take too long). The migration should just introduce an empty table and then we need to import it "on the side"
• One possible solution that came to my mind is fetching the configs using personal token (the script may run for a day or two but it shouldn't be a problem) and storing it in a format that can then be imported into the DB later.

[csomh]

> * to start with, we do not want to wait for projects to have a push/merge to main, we want to know the current state of configs without any user action so that we can find projects that need modifications.

Sorry if I missed something: why wouldn't we wait for user actions?

I thought the idea is to store the config on a per-pipeline basis, something like: when a GitHub event triggers a pipeline, the package config in that specific commit is saved in the pipeline and reused when subsequent events (Copr build progress, TF progress) move that pipeline forwards.

Apologies, wrong context 😄

[FrNecas]

Yup, I was about to write that we are probably talking about a different thing :D No worries ;)

[lbarcziova]

• on prod there are ~33,000 git projects enabled

😮
How did you come to this number?

[FrNecas]

SELECT COUNT(*) from git_projects on db dump from prod. Most of these won't have config enabled but we should probably check them anyway, right? Or maybe we could go based on projects which have pipelines attached, that would significantly reduce the number...

[lbarcziova]

Or maybe we could go based on projects which have pipelines attached, that would significantly reduce the number...

Agreed to do some kind of filtering (probably not all 33000 are really "relevant" projects).

[lachmanfrantisek]

Regarding the numbers, we have all the dist-git projects saved in the database. (Because of processing the messages from the Fedora-messaging bus..)

Here are the numbers since 2022-01-01 (https://prod.packit.dev/api/usage?from=2022-01-01 yes, it's slow and can timeout on the first run... but caching helps for the second try...;)

{
  "all_projects": {
    "project_count": 30328,
    "instances": {
      "salsa.debian.org": 1,
      "gitlab.gnome.org": 2,
      "src.fedoraproject.org": 22175,
      "git.centos.org": 25,
      "gitlab.freedesktop.org": 4,
      "gitlab.com": 9,
      "github.com": 8112
    }
  },
  "active_projects": {
    "project_count": 254,
    "top_projects_by_events_handled": SKIPPED
    "instances": {
      "github.com": 246,
      "gitlab.freedesktop.org": 3,
      "gitlab.com": 3,
      "gitlab.gnome.org": 2
    }
  },

Active projects = at least one pipeline in the time period.

---

edit: In the API output, there is a list of active projects that should have the config in a repo.

[lbarcziova]

With the script introduced in packit/research#176:

$ ./packit_config_checker.py list-affected migrations/tests_without_build_definition.py packit_configurations

Listing affected URLs from directory packit_configurations...
Affected repos (18): 
https://github.com/Dirout/depi
https://github.com/cockpit-project/cockpit-podman
https://github.com/pcahyna/tmt
https://github.com/martinpitt/cockpit-podman
https://github.com/Dirout/dokkoo
https://github.com/thrix/fmf
https://github.com/psss/python-nitrate
https://github.com/martinpitt/python-dbusmock
https://github.com/Dirout/ticky
https://github.com/cockpit-project/cockpit-machines
https://github.com/neoave/mrack
https://github.com/cockpit-project/starter-kit
https://github.com/martinpitt/cockpit-machines
https://github.com/fedora-iot/zezere
https://github.com/fedora-iot/greenboot
https://github.com/teemtee/fmf
https://github.com/Dirout/larz
https://github.com/cockpit-project/cockpit-certificates