Airflow 2.8.1 contains migration requiring SQLAlchemy 2+

[Tenzer]

Apache Airflow version

Other Airflow 2 version (please specify below)

If "Other Airflow 2 version" selected, which one?

2.8.1

What happened?

I've upgraded our Airflow installation to the newly released version 2.8.1 and ran airflow db upgrade, as prompted by the message when Airflow tries to start up.

That, however, resulted in the following stack trace:

[2024-01-19T16:06:41.883+0000] {migration.py:213} INFO - Context impl MySQLImpl.
[2024-01-19T16:06:41.886+0000] {migration.py:216} INFO - Will assume non-transactional DDL.
[2024-01-19T16:06:41.897+0000] {db.py:1616} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 10b52ebd31f7 -> 88344c1d9134, Drop unused TI index
Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airflow/__main__.py", line 57, in main
    args.func(args)
  File "/usr/local/lib/python3.11/site-packages/airflow/cli/cli_config.py", line 49, in command
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airflow/cli/commands/db_command.py", line 69, in upgradedb
    migratedb(args)
  File "/usr/local/lib/python3.11/site-packages/airflow/utils/cli.py", line 114, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airflow/utils/providers_configuration_loader.py", line 55, in wrapped_function
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airflow/cli/commands/db_command.py", line 129, in migratedb
    db.upgradedb(
  File "/usr/local/lib/python3.11/site-packages/airflow/utils/session.py", line 79, in wrapper
    return func(*args, session=session, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airflow/utils/db.py", line 1623, in upgradedb
    command.upgrade(config, revision=to_revision or "heads")
  File "/usr/local/lib/python3.11/site-packages/alembic/command.py", line 399, in upgrade
    script.run_env()
  File "/usr/local/lib/python3.11/site-packages/alembic/script/base.py", line 578, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/local/lib/python3.11/site-packages/alembic/util/pyfiles.py", line 93, in load_python_file
    module = load_module_py(module_id, path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/alembic/util/pyfiles.py", line 109, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/local/lib/python3.11/site-packages/airflow/migrations/env.py", line 117, in <module>
    run_migrations_online()
  File "/usr/local/lib/python3.11/site-packages/airflow/migrations/env.py", line 111, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/usr/local/lib/python3.11/site-packages/alembic/runtime/environment.py", line 937, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/usr/local/lib/python3.11/site-packages/alembic/runtime/migration.py", line 624, in run_migrations
    step.migration_fn(**kw)
  File "/usr/local/lib/python3.11/site-packages/airflow/migrations/versions/0133_2_8_1_refactor_dag_run_indexes.py", line 46, in upgrade
    op.drop_index("ti_state_incl_start_date", table_name="task_instance", if_exists=True)
  File "<string>", line 8, in drop_index
  File "<string>", line 3, in drop_index
  File "/usr/local/lib/python3.11/site-packages/alembic/operations/ops.py", line 1122, in drop_index
    return operations.invoke(op)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/alembic/operations/base.py", line 393, in invoke
    return fn(self, operation)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/alembic/operations/toimpl.py", line 113, in drop_index
    raise NotImplementedError("SQLAlchemy 2.0+ required")
NotImplementedError: SQLAlchemy 2.0+ required

What you think should happen instead?

Ideally, the migration should run without issues, using a version of SQLAlchemy that Airflow supports.

How to reproduce

Upgrade an existing Airflow installation to version 2.8.1. Run airflow db upgrade.

Operating System

Docker

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==8.16.0
apache-airflow-providers-celery==3.4.1
apache-airflow-providers-common-io==1.2.0
apache-airflow-providers-common-sql==1.7.2
apache-airflow-providers-ftp==3.5.2
apache-airflow-providers-google==10.13.1
apache-airflow-providers-http==4.5.2
apache-airflow-providers-imap==3.3.2
apache-airflow-providers-mysql==5.5.1
apache-airflow-providers-sqlite==3.4.3

Deployment

Other Docker-based deployment

Deployment details

No response

Anything else?

SQLAlchemy related packages:

Flask-SQLAlchemy==2.5.1
marshmallow-sqlalchemy==0.26.1
SQLAlchemy==1.4.49
sqlalchemy-bigquery==1.8.0
SQLAlchemy-JSONField==1.0.1.post0
sqlalchemy-redshift==0.8.14
sqlalchemy-spanner==1.6.2
SQLAlchemy-Utils==0.41.1

Alembic is at version 1.12.0.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[Tenzer]

I took a look at the Alembic changelog and realised 1.13.0 fixes a bug where some operations by mistake were marked as requiring SQLAlchemy 2.0+.

Upgrading our version of Alembic from 1.12.0 to the latest 1.13.1 version also fixes this problem.

Would it perhaps be a good idea to require the Alembic version to be at minimum version 1.13.0 so other people don't bump into this?

[Taragolis]

You need to install Airflow by provide constraints for specific Airflow and Python versions, for example in your case

pip install "apache-airflow==2.8.1” --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.8.1/constraints-3.11.txt"

This action will prevent surprises when upstream package is (not)updated and breaks some critical part of the Airflow.

airflow/constraints-3.11.txt

Lines 77 to 79 in ae9614d

	alabaster==0.7.16
	alembic==1.13.1
	alibabacloud-adb20211201==1.2.6

For the reference:

• Installation from PyPI
• [Reminder] How to reproducibly install Airflow

[Tenzer]

What a bizarre setup. I guess I'll have to see if I can somehow get that shoehorned into how we manage dependencies. Thank you for the pointer.

[potiuk]

What a bizarre setup. I guess I'll have to see if I can somehow get that shoehorned into how we manage dependencies. Thank you for the pointer,.

It's not bizarre, it's just. It is just designed for both - allowing for reproducible installation AND allowing people to upgrede depedencies when they neeed. In a few years a form of it will become a norm, simply the Python packaging and installation world did not yet feel then need for it, but with 670 dependencies of Airflow, this saves our users a lot of headaches, while allowing them also to use newer version of libraries. If they follow our advices that is.

If you want to know more about and have more context - I recommend to watch my talk from Airlfow Summit https://airflowsummit.org/sessions/2023/mastering-dependencies-the-airflow-way/

[potiuk]

BTW. yes. I agree, likely adding alembic min version in this case is a good idea as well. Alembic has very little dependencies on its own so this error is a good reason to add min-version for 2.8.2.