Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add "use self-hosted runners" to CI setup #38779

Merged
merged 6 commits into from
Apr 5, 2024
Merged

Add "use self-hosted runners" to CI setup #38779

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
7b0a34c
Add "use self-hosted runners" to CI setup
potiuk Apr 5, 2024
930df20
Update dev/breeze/doc/ci/04_selective_checks.md
potiuk Apr 5, 2024
402f834
Update dev/breeze/doc/ci/04_selective_checks.md
potiuk Apr 5, 2024
a08e3c9
Update dev/breeze/doc/ci/04_selective_checks.md
potiuk Apr 5, 2024
e3ad2a8
Update dev/breeze/doc/ci/04_selective_checks.md
potiuk Apr 5, 2024
b549288
Update dev/breeze/src/airflow_breeze/utils/selective_checks.py
potiuk Apr 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 39 additions & 29 deletions dev/breeze/doc/ci/04_selective_checks.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@
- [Suspended providers](#suspended-providers)
- [Selective check outputs](#selective-check-outputs)
- [Committer vs. non-committer PRs](#committer-vs-non-committer-prs)
- [PR labels](#pr-labels)
- [Changing behaviours of the CI runs by setting labels](#changing-behaviours-of-the-ci-runs-by-setting-labels)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

Expand Down Expand Up @@ -267,17 +267,29 @@ This is controlled by `Selective checks <04_selective_checks.md>`__ that set app
the build-info job of the workflow (see`is-committer-build` to `true`) if the actor is in the committer's
list and can be overridden by `non committer build` label in the PR.

Also, for most of the jobs, committer builds by default use "Self-hosted" runners, while non-committer
builds use "Public" runners. For committers, this can be overridden by setting the
`use public runners` label in the PR.
## Changing behaviours of the CI runs by setting labels

Also, currently for most of the jobs, committer builds by default use "Self-hosted" runners, while
non-committer builds use "Public" runners. For committers, this can be overridden by setting the
`use public runners` label in the PR. In the future when we might also switch committers to public runners.
Committers will be able to use `use self-hosted runners` label in the PR to force using self-hosted runners.
The `use public runners` label will still be available for committers and they will be able to set it for
builds that also have `canary` label set to also switch the `canary` builds to public runners.

If you are testing CI workflow changes and want to test it for more complete matrix combinations generated by
the jobs - you can set `all versions` label in the PR. This will run the PRs with the same combinations
of versions as the `canary` main build. Using `all versions` is automatically set when build or project
dependencies change in `pyproject.toml`.

If you are testing CI workflow changes and change `pyproject.toml` or `generated/provider_dependencies.json` and you
want to limit the number of matrix combinations generated by
of versions as the `canary` main build. Using `all versions` is automatically set when build dependencies
change in `pyproject.toml` or when dependencies change for providers in `generated/provider_dependencies.json`
or when `hatch_build.py` changes.

If you are running an `apache` PR, you can also set `canary` label for such PR and in this case, all the
`canary` properties of build will be used: `self-hosted` runners, `full tests needed` mode, `all versions`
as well as all canary-specific jobs will run there. You can modify this behaviour of the `canary` run by
applying `use public runners`, and `default versions only` labels to the PR as well which will still run
a `canary` equivalent build but with public runners an default Python/K8S versions only - respectively.

If you are testing CI workflow changes and change `pyproject.toml` or `generated/provider_dependencies.json`
and you want to limit the number of matrix combinations generated by
the jobs - you can set `default versions only` label in the PR. This will limit the number of versions
used in the matrix to the default ones (default Python version and default Kubernetes version).

Expand All @@ -288,30 +300,28 @@ used in the matrix to the latest ones (latest Python version and latest Kubernet
You can also disable cache if you want to make sure your tests will run with image that does not have
left-over package installed from the past cached image - by setting `disable image cache` label in the PR.

By default all outputs of successful parallel tests are not shown. You can enable them by setting
By default, all outputs of successful parallel tests are not shown. You can enable them by setting
`include success outputs` label in the PR. This makes the logs of mostly successful tests a lot longer
and more difficult to sift through, but it might be useful in case you want to compare successful and
unsuccessful runs of the tests.

## PR labels

As mentioned below, you can influence the outputs of selected checks by setting labels to the PR. Here is
am overview of possible labels and their meaning:

| Label | Affected outputs | Meaning |
|-------------------------------|-------------------------------|-----------------------------------------------------------------------------------------------------------------|
| canary | is-canary-run | If set, the PR run from apache/airflow repo behaves as `canary` run (can only be run by maintainer). |
| debug ci resources | debug-ci-resources | If set, then debugging resources is enabled during parallel tests and you can see them in the output. |
| default versions only | all-versions, *-versions-* | If set, the number of Python and Kubernetes, DB versions used by the build will be limited to the default ones. |
| disable image cache | docker-cache | If set, the image cache is disables when building the image. |
| include success outputs | include-success-outputs | By default, outputs of successful parallel tests are not shown - enabling this flag will make then shown. |
| latest versions only | *-versions-*, *-versions-* | If set, the number of Python, Kubernetes, DB versions used by the build will be limited to the latest ones. |
| all versions | all-versions, *-versions-* | Run tests for all python and k8s versions. |
| full tests needed | full-tests-needed | Run complete set of tests (might be with default or all python/k8s versions) |
| non committer build | is-committer-build | If set then even for non-committer builds, the scripts used for images are used from target branch. |
| upgrade to newer dependencies | upgrade-to-newer-dependencies | If set to true (default false) then dependencies in the CI image build are upgraded to the newer ones. |
| use public runners | runs-on-as-json-default | Force using public runners even for Committer runs. |

This table summarizes the labels you can use on PRs to control the selective checks and the CI runs:

| Label | Affected outputs | Meaning |
|----------------------------------|----------------------------------|-------------------------------------------------------------------------------------------|
| all versions | all-versions, *-versions-* | Run tests for all python and k8s versions. |
| allow suspended provider changes | allow-suspended-provider-changes | Allow changes to suspended providers. |
| canary | is-canary-run | If set, the PR run from apache/airflow repo behaves as `canary` run. |
| debug ci resources | debug-ci-resources | If set, then debugging resources is enabled during parallel tests and you can see them. |
| default versions only | all-versions, *-versions-* | If set, the number of Python and Kubernetes, DB versions are limited to the default ones. |
| disable image cache | docker-cache | If set, the image cache is disables when building the image. |
| full tests needed | full-tests-needed | If set, complete set of tests are run |
| include success outputs | include-success-outputs | If set, outputs of successful parallel tests are shown not only failed outputs. |
| latest versions only | *-versions-*, *-versions-* | If set, the number of Python, Kubernetes, DB versions will be limited to the latest ones. |
| non committer build | is-committer-build | If set, the scripts used for images are used from target branch for committers. |
| upgrade to newer dependencies | upgrade-to-newer-dependencies | If set to true (default false) then dependencies in the CI image build are upgraded. |
| use public runners | runs-on-as-json-default | Force using public runners as default runners. |
| use self-hosted runners | runs-on-as-json-default | Force using self-hosted runners as default runners. |

-----

Expand Down
32 changes: 26 additions & 6 deletions dev/breeze/src/airflow_breeze/utils/selective_checks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,16 +63,17 @@
from airflow_breeze.utils.provider_dependencies import DEPENDENCIES, get_related_providers
from airflow_breeze.utils.run_utils import run_command

FULL_TESTS_NEEDED_LABEL = "full tests needed"
ALL_VERSIONS_LABEL = "all versions"
DEBUG_CI_RESOURCES_LABEL = "debug ci resources"
USE_PUBLIC_RUNNERS_LABEL = "use public runners"
NON_COMMITTER_BUILD_LABEL = "non committer build"
DEFAULT_VERSIONS_ONLY_LABEL = "default versions only"
ALL_VERSIONS_LABEL = "all versions"
LATEST_VERSIONS_ONLY_LABEL = "latest versions only"
DISABLE_IMAGE_CACHE_LABEL = "disable image cache"
FULL_TESTS_NEEDED_LABEL = "full tests needed"
INCLUDE_SUCCESS_OUTPUTS_LABEL = "include success outputs"
LATEST_VERSIONS_ONLY_LABEL = "latest versions only"
NON_COMMITTER_BUILD_LABEL = "non committer build"
UPGRADE_TO_NEWER_DEPENDENCIES_LABEL = "upgrade to newer dependencies"
USE_PUBLIC_RUNNERS_LABEL = "use public runners"
USE_SELF_HOSTED_RUNNERS_LABEL = "use self-hosted runners"


ALL_CI_SELECTIVE_TEST_TYPES = (
Expand Down Expand Up @@ -1114,7 +1115,11 @@ def affected_providers_list_as_string(self) -> str | None:
def runs_on_as_json_default(self) -> str:
if self._github_repository == APACHE_AIRFLOW_GITHUB_REPOSITORY:
if self._github_event in [GithubEvents.SCHEDULE, GithubEvents.PUSH]:
# Canary and Scheduled runs
return RUNS_ON_SELF_HOSTED_RUNNER
if self._pr_labels and USE_PUBLIC_RUNNERS_LABEL in self._pr_labels:
# Forced public runners
return RUNS_ON_PUBLIC_RUNNER
actor = self._github_actor
if self._github_event in (GithubEvents.PULL_REQUEST, GithubEvents.PULL_REQUEST_TARGET):
try:
Expand All @@ -1129,8 +1134,23 @@ def runs_on_as_json_default(self) -> str:
f"[info]Could not find the actor from pull request, "
f"falling back to the actor who triggered the PR: {actor}[/]"
)
if actor in COMMITTERS and USE_PUBLIC_RUNNERS_LABEL not in self._pr_labels:
if (
actor not in COMMITTERS
and self._pr_labels
and USE_SELF_HOSTED_RUNNERS_LABEL in self._pr_labels
):
get_console().print(
f"[error]The PR has `{USE_SELF_HOSTED_RUNNERS_LABEL}` label, but "
f"{actor} is not a committer. This is not going to work.[/]"
)
sys.exit(1)
if USE_SELF_HOSTED_RUNNERS_LABEL in self._pr_labels:
# Forced self-hosted runners
return RUNS_ON_SELF_HOSTED_RUNNER
if actor in COMMITTERS:
return RUNS_ON_SELF_HOSTED_RUNNER
else:
return RUNS_ON_PUBLIC_RUNNER
return RUNS_ON_PUBLIC_RUNNER

@cached_property
Expand Down
Loading