migrateDatabaseJob fails when upgrading to 2.6.1 with Sentry enabled

[gil-tober]

Apache Airflow version

2.6.1

What happened

When upgrading Airflow from 2.5.3 to 2.6.1 the airflow-run-migration job fails.

I have deployed the same setup to our DEV environment (where Sentry is disabled) and the upgrade succeeded without an issue.

After that I tried deploying to our PROD environment where Sentry is enabled and the job failed with the error below.

I tried enabling Sentry on DEV and the upgrade failed.

Took a look at airflow.executors.executor_loader.py at lines 162-185 when the exception is raised from. Setting the environment variable _AIRFLOW__SKIP_DATABASE_EXECUTOR_COMPATIBILITY_CHECK=1 fixed the error getting in the upgrade job. (Our metadb is Postgres)

Traceback (most recent call last):
  File “/home/airflow/.local/bin/airflow”, line 5, in <module>
    from airflow.__main__ import main
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/__init__.py”, line 66, in <module>
    settings.initialize()
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/settings.py”, line 522, in initialize
    import_local_settings()
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/settings.py”, line 467, in import_local_settings
    import airflow_local_settings
  File “/opt/airflow/dags/repo/airflow_local_settings.py”, line 5, in <module>
    from airflow.www.utils import UIAlert
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/www/utils.py”, line 44, in <module>
    from airflow.models.dagrun import DagRun
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/models/dagrun.py”, line 57, in <module>
    from airflow.models.taskinstance import TaskInstance as TI
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/models/taskinstance.py”, line 100, in <module>
    from airflow.sentry import Sentry
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/sentry.py”, line 195, in <module>
    Sentry = ConfiguredSentry()
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/sentry.py”, line 92, in __init__
    executor_class, _ = ExecutorLoader.import_default_executor_cls()
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/executors/executor_loader.py”, line 158, in import_default_executor_cls
    executor, source = cls.import_executor_cls(executor_name)
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/executors/executor_loader.py”, line 134, in import_executor_cls
    return _import_and_validate(cls.executors[executor_name]), ConnectorSource.CORE
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/executors/executor_loader.py”, line 130, in _import_and_validate
    cls.validate_database_executor_compatibility(executor)
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/executors/executor_loader.py”, line 181, in validate_database_executor_compatibility
    from airflow.settings import engine
ImportError: cannot import name ‘engine’ from ‘airflow.settings’ (/home/airflow/.local/lib/python3.10/site-packages/airflow/settings.py)

What you think should happen instead

migrateDatabaseJob should not fail when Sentry is enabled

How to reproduce

Upgrade from 2.5.3 to 2.6.1 with CeleryKubernetesExecutor and Sentry enabled

Operating System

Debian GNU/Linux 11 (bullseye)

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==8.0.0
apache-airflow-providers-celery==3.1.0
apache-airflow-providers-cncf-kubernetes==6.1.0
apache-airflow-providers-common-sql==1.4.0
apache-airflow-providers-docker==3.6.0
apache-airflow-providers-elasticsearch==4.4.0
apache-airflow-providers-ftp==3.3.1
apache-airflow-providers-google==8.3.0
apache-airflow-providers-grpc==3.1.0
apache-airflow-providers-hashicorp==3.3.1
apache-airflow-providers-http==4.3.0
apache-airflow-providers-imap==3.1.1
apache-airflow-providers-jenkins==3.2.1
apache-airflow-providers-microsoft-azure==6.0.0
apache-airflow-providers-mysql==5.0.0
apache-airflow-providers-odbc==3.2.1
apache-airflow-providers-postgres==5.3.1
apache-airflow-providers-redis==3.1.0
apache-airflow-providers-salesforce==5.3.0
apache-airflow-providers-sendgrid==3.1.0
apache-airflow-providers-sftp==4.2.4
apache-airflow-providers-slack==7.2.0
apache-airflow-providers-snowflake==4.0.5
apache-airflow-providers-sqlite==3.3.2
apache-airflow-providers-ssh==3.6.0
apache-airflow-providers-tableau==4.1.0

Deployment

Official Apache Airflow Helm Chart

Deployment details

Airflow deployed on an EKS cluster using the official Airflow Helm Chart (V1.9.0)
MetaBD is a Postgres RDS
CeleryKubernetesExecutor

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[hussein-awala]

Probably it's a circular import issue similar to what we had in #31033.

[gil-tober]

Hey @hussein-awala
I saw #31033, the import of Engine in the error that i'm getting is referencing an sqlalchemy.engine.base object.
It could be that there is a circular import there, but first, there isn't a exception for that and second, I'm not that deeply familiar with airflow.settings.py to see it.

[potiuk]

Your problem is likely this:

  File “/opt/airflow/dags/repo/airflow_local_settings.py”, line 5, in <module>
    from airflow.www.utils import UIAlert

Airflow local seetings is loaded during initialization and for some reason, you are trying to import UIAlert there, which in turn tries to (through chain of imports) import settings again, but local_settins are imported during importing settings, so when you agan try to import settings (that are partially imported), Python (rightfully) refuses to cooperate as it would cause circula import and recurrrent operations (settings would again import local_settings which would then again import settings and so-one.

So the question is - why are you trying to import UIAlert in the top level of your local_settings ? Is it really necessary ? Can you change it to local import instead inside a function which you have in your local_settings so that it is not imported when airflow_local_settings is parsed ?

[gil-tober]

Hey @potiuk
The implemantaion was made based on Airflow's docs and there is no note about global level import.

On Sunday I will try to refactor airflow_local_settings.py to do local imports and will post my results.

I will say, as stated in the initial issue, deploying the same code base to our dev environment where AIRFLOW__SENTRY__SENTRY_ON=False doesn't cause this issue.
I also did the check of deploying to dev (Sentry disabled) and re-deploying after enabling Sentry, and the second deployment failed while the first succeeded

[gil-tober]

Hey @potiuk

I'm trying to understand how should I add alerts/messages to the UI
Looking at airflow.settings.py

airflow/airflow/settings.py

Lines 488 to 495 in 9137740

	for name in names:
	# If we have already handled a function by adding it to the plugin, then don't clobber the global
	# function
	if name in plugin_functions:
	continue
	globals()[name] = getattr(airflow_local_settings, name)

From what I understand airflow_local_settings.py should have an DASHBOARD_UIALERTS variable in order to overwrite the global dashboard alerts.

---

I first tried

def dashboard_ui_alerts():
    from airflow.www.utils import UIAlert
    return [UIAlert("<strong>Welcome to Airflow!</strong>", html=True)]


dashboard_ui_alerts()

And as i thought, the message was not rendered in the UI as it is not setting globals()["DASHBOARD_UIALERTS"]

---

After that I tried

def dashboard_ui_alerts():
    from airflow.www.utils import UIAlert
    return [UIAlert("<strong>Welcome to Airflow!</strong>", html=True)]


DASHBOARD_UIALERTS = dashboard_ui_alerts()

And I'm still getting the same error (a bit different traceback because the import of UIAlert is local to dashboard_ui_alerts

Traceback (most recent call last):
  File “/home/airflow/.local/bin/airflow”, line 5, in <module>
    from airflow.__main__ import main
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/__init__.py”, line 66, in <module>
    settings.initialize()
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/settings.py”, line 522, in initialize
    import_local_settings()
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/settings.py”, line 467, in import_local_settings
    import airflow_local_settings
  File “/opt/airflow/dags/repo/airflow_local_settings.py”, line 120, in <module>
    DASHBOARD_UIALERTS = render_ui_alerts()
  File “/opt/airflow/dags/repo/airflow_local_settings.py”, line 92, in render_ui_alerts
    from airflow.www.utils import UIAlert
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/www/utils.py”, line 44, in <module>
    from airflow.models.dagrun import DagRun
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/models/dagrun.py”, line 57, in <module>
    from airflow.models.taskinstance import TaskInstance as TI
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/models/taskinstance.py”, line 100, in <module>
    from airflow.sentry import Sentry
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/sentry.py”, line 195, in <module>
    Sentry = ConfiguredSentry()
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/sentry.py”, line 92, in __init__
    executor_class, _ = ExecutorLoader.import_default_executor_cls()
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/executors/executor_loader.py”, line 158, in import_default_executor_cls
    executor, source = cls.import_executor_cls(executor_name)
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/executors/executor_loader.py”, line 134, in import_executor_cls
    return _import_and_validate(cls.executors[executor_name]), ConnectorSource.CORE
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/executors/executor_loader.py”, line 130, in _import_and_validate
    cls.validate_database_executor_compatibility(executor)
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/executors/executor_loader.py”, line 181, in validate_database_executor_compatibility
    from airflow.settings import engine
ImportError: cannot import name ‘engine’ from ‘airflow.settings’ (/home/airflow/.local/lib/python3.10/site-packages/airflow/settings.py)

I'll mentioned again that this error only occurs when AIRFLOW__SENTRY__SENTRY_ON = True

Thanks for all the help

[potiuk]

Indeed. Seems that adding UI Alert this way won't work with sentry. This is a bug. Converted it into a bug (the root cause is explained there):

#31442

As a workaround - not sure if this will work but maybe you could create the Welcome message UI alert in on_starting listener during starting of SchedulerJob. I am not 100% sure if it is going to work, but It might be a solution for you for now (and actually if we find out that it works, It might be a good idea to add it as a recommended way of creating the alert in case of using sentry.

[gil-tober]

Thanks for the quick response

I will try the workaround you suggested and will post my findings.

For now, I found that setting _AIRFLOW__SKIP_DATABASE_EXECUTOR_COMPATIBILITY_CHECK=1 also works as workaround.

airflow/airflow/executors/executor_loader.py

Lines 160 to 186 in 9137740

	@classmethod
	@functools.lru_cache(maxsize=None)
	def validate_database_executor_compatibility(cls, executor: type[BaseExecutor]) -> None:
	"""Validate database and executor compatibility.
	Most of the databases work universally, but SQLite can only work with
	single-threaded executors (e.g. Sequential).
	This is NOT done in ``airflow.configuration`` (when configuration is
	initialized) because loading the executor class is heavy work we want to
	avoid unless needed.
	"""
	# Single threaded executors can run with any backend.
	if executor.is_single_threaded:
	return
	# This is set in tests when we want to be able to use SQLite.
	if os.environ.get("_AIRFLOW__SKIP_DATABASE_EXECUTOR_COMPATIBILITY_CHECK") == "1":
	return
	from airflow.settings import engine
	# SQLite only works with single threaded executors
	if engine.dialect.name == "sqlite":
	raise AirflowConfigException(f"error: cannot use SQLite with the {executor.__name__}")

Our metadb is Postgres, so the check for sqlite is not relevant for us

[potiuk]

Nice workaround too :)

[potiuk]

I am turning that into a discussion, because it is evidently some troubleshooting of mis-configured local settings.