-
Notifications
You must be signed in to change notification settings - Fork 14.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Version 2.3.3 breaks "Plugins as Python packages" feature #25271
Comments
Thanks for opening your first issue here! Be sure to follow the issue template! |
Yeah. It would be great to get to the bottom of this. Happy to help with investigation if you have some more findings. |
In FAB 2.X, registering the same blueprint more than once is prohibited. https://github.com/pallets/flask/blob/2.1.2/src/flask/blueprints.py#L293-L296 Problem is We need a way to prevents double registration for plugins.. |
The most likely cause is the plugin appears twice in |
Indeed. One reason this can happen is if there are separate For example:
|
It’s probably not pip installing things to both directories; on some systems these two are symbolically linked (i.e. the same directory). |
I have very similar issue with duplicate plugins for
Any ideas why I have 2 entries in |
Small update: Seems like above issue happens for me only during running |
Apache Airflow version
2.3.3 (latest released)
What happened
In 2.3.3
If I use https://airflow.apache.org/docs/apache-airflow/stable/plugins.html#plugins-as-python-packages feature, then I see these Error:
short:
ValueError: The name 'airs' is already registered for this blueprint. Use 'name=' to provide a unique name.
long:
If I don't use it(workarounding by AIRFLOW__CORE__PLUGINS_FOLDER), errors doesn't occur.
It didn't happend in 2.3.2 and earlier
What you think should happen instead
Looks like plugins are import multiple times if it is plugins-as-python-packages.
Perhaps flask's major version change is the main cause.
Presumably, in flask 1.0, duplicate registration of blueprint was quietly filtered out, but in 2.0 it seems to have been changed to generate an error. (I am trying to find out if this hypothesis is correct)
Anyway, use the latest version of FAB is important. we will have to adapt to this change, so plugins will have to be imported once regardless how it defined.
How to reproduce
Operating System
CentOS Linux release 7.9.2009 (Core)
Versions of Apache Airflow Providers
$ SHIV_INTERPRETER=1 airsflow -m pip freeze | grep apache- apache-airflow==2.3.3 apache-airflow-providers-apache-hive==3.1.0 apache-airflow-providers-apache-spark==2.1.0 apache-airflow-providers-celery==3.0.0 apache-airflow-providers-common-sql==1.0.0 apache-airflow-providers-ftp==3.1.0 apache-airflow-providers-http==3.0.0 apache-airflow-providers-imap==3.0.0 apache-airflow-providers-postgres==5.1.0 apache-airflow-providers-redis==3.0.0 apache-airflow-providers-sqlite==3.1.0
but I think these are irrelevant.
Deployment
Other 3rd-party Helm chart
Deployment details
docker image based on centos7, python 3.9.10 interpreter, self-written helm2 chart ....
... but I think these are irrelevant.
Anything else
No response
Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: