Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kubernetes executor can adopt tasks from other schedulers #10996

Merged
merged 4 commits into from
Oct 1, 2020

Conversation

dimberman
Copy link
Contributor

@dimberman dimberman commented Sep 17, 2020

Since Kubernetes allows dynamic changing of labels for k8s pods, we can modify labels s.t.
a KubeJobWatcher can monitor tasks launched by another scheduler.


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

@boring-cyborg boring-cyborg bot added area:CLI area:Scheduler including HA (high availability) scheduler k8s labels Sep 17, 2020
airflow/cli/commands/dag_command.py Outdated Show resolved Hide resolved
airflow/executors/base_executor.py Outdated Show resolved Hide resolved
@dimberman dimberman force-pushed the kubernetes-executor-ha branch 2 times, most recently from bf9e030 to 1c29d60 Compare September 21, 2020 14:50
@dimberman dimberman force-pushed the kubernetes-executor-ha branch 3 times, most recently from 8b7e691 to 542d03c Compare September 23, 2020 00:52
@dimberman dimberman changed the title Kubernetes executor ha Kubernetes executor can adopt tasks from other schedulers Sep 23, 2020
@dimberman dimberman force-pushed the kubernetes-executor-ha branch from 42977a5 to 6bb369e Compare September 23, 2020 18:19
@dimberman dimberman force-pushed the kubernetes-executor-ha branch 4 times, most recently from b915807 to 75e4795 Compare September 27, 2020 14:35
ashb
ashb previously requested changes Sep 28, 2020
airflow/executors/base_executor.py Outdated Show resolved Hide resolved
airflow/executors/base_executor.py Outdated Show resolved Hide resolved
airflow/executors/kubernetes_executor.py Outdated Show resolved Hide resolved
airflow/executors/kubernetes_executor.py Outdated Show resolved Hide resolved
airflow/executors/kubernetes_executor.py Outdated Show resolved Hide resolved
@dimberman dimberman force-pushed the kubernetes-executor-ha branch from 75e4795 to 254d073 Compare October 1, 2020 16:33
@dimberman dimberman force-pushed the kubernetes-executor-ha branch from ec0a268 to 7900576 Compare October 1, 2020 17:16
@dimberman dimberman requested a review from ashb October 1, 2020 17:19
@dimberman dimberman dismissed ashb’s stale review October 1, 2020 17:28

all issues addressed

@ryw
Copy link
Member

ryw commented Oct 1, 2020

@dimberman is there some documentation that should be updated w/ this change?

@dimberman dimberman merged commit 3ca11eb into apache:master Oct 1, 2020
@dimberman dimberman deleted the kubernetes-executor-ha branch October 1, 2020 19:07
@dimberman
Copy link
Contributor Author

@ryw I don't think further docs needed as this isn't intended to be used externally.

potiuk added a commit that referenced this pull request Nov 15, 2020
(cherry picked from commit 3ca11eb)

Co-authored-by: Daniel Imberman <[email protected]>
@potiuk potiuk added this to the Airflow 1.10.13 milestone Nov 15, 2020
@potiuk potiuk added the type:improvement Changelog: Improvements label Nov 15, 2020
potiuk added a commit that referenced this pull request Nov 16, 2020
(cherry picked from commit 3ca11eb)

Co-authored-by: Daniel Imberman <[email protected]>
potiuk added a commit that referenced this pull request Nov 16, 2020
(cherry picked from commit 3ca11eb)

Co-authored-by: Daniel Imberman <[email protected]>
@kaxil kaxil added provider:cncf-kubernetes Kubernetes provider related issues and removed area:k8s labels Nov 18, 2020
@kaxil kaxil removed this from the Airflow 1.10.13 milestone Nov 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:CLI area:Scheduler including HA (high availability) scheduler provider:cncf-kubernetes Kubernetes provider related issues type:improvement Changelog: Improvements
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants