Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove the ability to import executors from plugins #43289

Merged
merged 1 commit into from
Oct 23, 2024

Conversation

kaxil
Copy link
Member

@kaxil kaxil commented Oct 23, 2024

Executors should no longer be registered or imported via Airflow's plugin mechanism -- these types of classes are just treated as plain python classes by Airflow, so there is no need to register them with Airflow.

We had already removed it from our docs: https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/plugins.html#plugins


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added area:Executors-core LocalExecutor & SequentialExecutor area:plugins area:webserver Webserver related Issues labels Oct 23, 2024
@kaxil kaxil added the airflow3.0:breaking Candidates for Airflow 3.0 that contain breaking changes label Oct 23, 2024
@kaxil kaxil added this to the Airflow 3.0.0 milestone Oct 23, 2024
@kaxil kaxil requested a review from Lee-W October 23, 2024 01:38
@kaxil kaxil force-pushed the remove-executor-support-from-plugin branch from 71ccbc7 to 1e673d6 Compare October 23, 2024 01:54
Copy link
Contributor

@jscheffl jscheffl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we have a still ongoing discussion, not voted towards future scope of provders package. I think this discussion should settle first.

It would also mean that if this PR is merged, that the executor descriptions in provider packages need to be removed - or at least be deprecated until Airflow 2.x support is generated.

@kaxil
Copy link
Member Author

kaxil commented Oct 23, 2024

I think we have a still ongoing discussion, not voted towards future scope of provders package. I think this discussion should settle first.

It would also mean that if this PR is merged, that the executor descriptions in provider packages need to be removed - or at least be deprecated until Airflow 2.x support is generated.

@jscheffl The change in this PR just impact importing executors "via plugins": https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/plugins.html#plugins

It is not removing executors from Providers

@kaxil kaxil force-pushed the remove-executor-support-from-plugin branch from 1e673d6 to 5bb9663 Compare October 23, 2024 11:52
@kaxil kaxil requested a review from ephraimbuddy as a code owner October 23, 2024 11:52
Executors should no longer be registered or imported via Airflow's plugin mechanism -- these types of classes
are just treated as plain python classes by Airflow, so there is no need to register them with Airflow.

# Conflicts:
#	tests/plugins/test_plugin.py
@kaxil kaxil force-pushed the remove-executor-support-from-plugin branch from 5bb9663 to 768bb3c Compare October 23, 2024 13:04
@kaxil kaxil merged commit 9da6fc3 into apache:main Oct 23, 2024
82 checks passed
@kaxil kaxil deleted the remove-executor-support-from-plugin branch October 23, 2024 14:37
harjeevanmaan pushed a commit to harjeevanmaan/airflow that referenced this pull request Oct 23, 2024
Executors should no longer be registered or imported via Airflow's plugin mechanism -- these types of classes
are just treated as plain python classes by Airflow, so there is no need to register them with Airflow.
PaulKobow7536 pushed a commit to PaulKobow7536/airflow that referenced this pull request Oct 24, 2024
Executors should no longer be registered or imported via Airflow's plugin mechanism -- these types of classes
are just treated as plain python classes by Airflow, so there is no need to register them with Airflow.
ellisms pushed a commit to ellisms/airflow that referenced this pull request Nov 13, 2024
Executors should no longer be registered or imported via Airflow's plugin mechanism -- these types of classes
are just treated as plain python classes by Airflow, so there is no need to register them with Airflow.
@Lee-W Lee-W mentioned this pull request Nov 18, 2024
96 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
airflow3.0:breaking Candidates for Airflow 3.0 that contain breaking changes area:Executors-core LocalExecutor & SequentialExecutor area:plugins area:webserver Webserver related Issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants