Replace SQLAlchemy Migrate with Alembic

[jdavcs]

NOTE: This branch has been superseded by #13513. However, the description of the migration system provided with this PR is accurate and can be used as implementation reference for #13513.

---

Ref #10369.
Supersedes #12869.

This is a clean branch with a cleaned-up commit history; some commits are large, but all changes are logically grouped together. (the commit history in #12869 shows most of the previous iterations)

Migrations system overview

Galaxy's data model is split into the galaxy model and the install model. These two models may be persisted in one combined database or two separate databases. To accommodate this setup, as well as a seamless transition from a combined database to separate databases and vice-versa, the proposed new Alembic-based migrations system is built on top of branches.

A branch, in this context, is a versioning lineage (or migration stream) that starts at a common root (base revision) and represents some part of the data model. These branches are permanent, are identified by labels, and may share the same Alembic version table (if they share the same database; otherwise, each database has its own version table).

We use 2 branches: gxy (galaxy) and tsi (tool shed install). These IDs were selected for their brevity and visual consistency: they are referenced throughout the system as version directory names and branch labels (by Alembic), and model identifiers. In addition, the term "install" in "install model" may be ambiguous in some contexts (it's easy to mistake it for a verb, etc.).

Each branch has its own version history, represented by revision scripts located in the branch version directory (e.g. revisions for the galaxy model are located in the gxy version directory, whereas revisions for the install model are located in the tsi version directory). The version directories contain labeled base revisions, which were created with the corresponding branch labels. When we create a new revision, we only need to indicate the branch label: the system will know where to place the new revision file.

Alembic requires its own configuration file (alembic.ini); we use it for required settings only: the location of the alembic script and the version directories, and the default Alembic logging configuration. All other configuration information, including how the two models are persisted, is programmatically set at run time based on Galaxy's own configuration, which remains the one source of truth. There are no changes to how we indicate separate vs. combined database storage: if the install_database_connection option is set to a value other than the value of database_connection, two databases will be used instead of a combined database. If the setting is changed for an existing database, our system will know how to handle this (see tests for such scenarios).

See relevant Alembic documentation:
https://alembic.sqlalchemy.org/en/latest/branches.html#working-with-branches
and especially here:
https://alembic.sqlalchemy.org/en/latest/branches.html#working-with-explicit-branches
https://alembic.sqlalchemy.org/en/latest/branches.html#working-with-branch-labels
https://alembic.sqlalchemy.org/en/latest/branches.html#working-with-multiple-bases

Directory structure

lib/galaxy/model/migrations/
├── alembic
│   ├── versions_gxy
│   │   └── e7b6dcb09efd_create_gxy_branch.py
│   ├── versions_tsi
│   │   └── d4a650f47a3c_create_tsi_branch.py
│   ├── env.py
│   └── script.py.mako
├── README.md
├── __init__.py
├── alembic.ini
└── scripts.py

versions_*: version directories, containing revision scripts; filenames include a GUID-based revision identifier, assigned by Alembic. Ordering is set via directives inside the revision scripts (down_revision)

env.py: invoked by Alembic whenever a command is executed. Its primary purpose is to determine the target database (based on the indicated branch) and configure the migration environment.

script.py.mako: the template file used to generate revisions.

README.md: basic user documentation on how to use the new migrations system.

__init__.py: system core (see Main classes and entry points).

alembic.ini: Alembic configuration file

scripts.py: helpers used for invoking Alembic via script (create_db/migrate_db)

Main classes and entry points

• verify_databases, verify_databases_via_script: entry points

• AlembicManager: handles Alembic interactions with one database; serves as a facade to a subset of Alembic commands.

• DatabaseStateCache: holds a snapshot of database state. This is an abstraction layer that simplifies the code and reduces (1 vs. 4) the number of calls to the database.

• DatabaseStateVerifier: executes the core migration algorithm.

Migration algorithm

When invoked via create_db.sh script or as part of Galaxy's startup process (in config/__init__.py), the system follows the same algorithm. (Unlike create_db.sh, the migrate_db.sh script is a thin wrapper around Alembic, so it does not verify the database state; it determines the target branch, retrieves relevant configuration values and invokes Alembic.)

We start at one of the two entry points; then, subsequently, create and run a DatabaseStateVerifier for the gxy model, then for the tsi model. The system checks if the database exists and is non-empty, and then determines whether it is up-to-date, and if not, whether it can be upgraded automatically. The algorithm tries to minimize database calls by using a cache (DatabaseStateCache) and by taking shortcuts when possible (e.g. is_new_database will be set if while processing the first model a new or an empty database was initialized).

The algorithm makes "an educated guess" only in one case. If within an existing, nonempty database that is combined, the gxy model is up-to-date, but the tsi model is not, we cannot determine the state of the tsi model tables with absolute certainty. Such an edge might happen for several reasons (see inline comments in TestPartiallyUpgradedCombinedDatabase for a detailed description). We cannot determine the state of this tsi model because we do not store the SQLAlchemy Migrate version of the install model in a combined database. So, the reasoning goes like this: if there are
no tsi tables in the database, we simply treat it as a new install; but if there is at least one table, we assume it is the same version as gxy (if gxy was manually upgraded, we assume the same for tsi). I think, there's nothing else we can reasonably do, other than fail unconditionally, which seems unreasonable in this case.

The diagram below describes the algorithm.

(UPDATED on 8/21/22: same logic; better image from GCC 2022 talk)

(UPDATED ON 2/18/22; see changes to the paths "has alembic table? > no" and "is sqlalchemy migrate version latest? > yes")

Previous version:

Misc. implementation notes:

• Old migration scripts have been discarded: we either start from an empty database, or from a database that has been upgraded to the last SQLAlchemy Migration (version 180), at which point an upgrade can be ran automatically.

• We don't automatically upgrade the database; we do so only if the database_auto_migrate option has been set`. Otherwise, the systems raises an error with an informative message.

• SQLAlchemy Migrate has not been deleted from Galaxy's Python dependencies list because it is still used for Tool Shed. However, SQLAlchemy Migrate is not compatible with SQLAlchemy 2.0, so moving Tool Shed to Alembic as well seems a reasonable step, given Galaxy's upcoming migration to SQLAlchemy 2.0. Meanwhile, as a temporary fix, I have created TS-specific scripts to create and migrate the Tool Shed databse using SQLAlchemy Migrate.

• The GALAXY_ALLOW_FUTURE_DATABASE setting has been removed. Revision ordering is no longer determined by a sequence of ascending integers, so determining whether a given revision head stored in the database is a "future" version is no longer possible (if it's more than one version removed from the head in the code). Since this was used for development instances only, an alternative solution is to simply set check_migrate_databases=False. Another alternative is to keep the new revision scripts in the versions directories, so that the system can identify the head stored in the database as a valid revision. Although, I think, if one is knowingly running a database whose version is out of sync with the code base, verifying the database state programmatically makes little sense, so just unset check_migrate_databases.

Discarded approaches

I considered multiple alternatives, primarily these:

• Use 2 alembic directories (2 sets of everything)
• Use 2 migration environments (galaxy and install), use --name argument to distiguish between the two
  (ref: https://alembic.sqlalchemy.org/en/latest/cookbook.html#run-multiple-alembic-environments-from-one-ini-file)
• Variations of the above

Reasons for discarding (apply to some or all of the discarded options):

• Applicable to completely independent lineages: not the case with galaxy/install
• Impossible to set dependencies between branches: we might need that
• Lots of duplicate code: we only need one env.py and one mako template (with 2 alembic directories, we get duplicate alembic.ini as well)
• Can't run without the --name argument, so for a combined database would need a 3 values: galaxy, install, both - which doesn't feel right for multiple reasons
• Main reason: not suitable for our core use case: although we have 2 distinct models, they may be mapped to one OR two databases, and we must accommodate both. Furthermore, setups may change: 2 >> 1 and 1 >> 2. I ran into multiple issues with this point, and was unable to find a reasonably clean implementation. The main issue was targeting commands at one or the other or both with different database setups.

Discarded migration algorithm

In the current version, the gxy and tsi models are verified in sequence. In a previous version, they were verified at the same time. That saved a few trips to the database, but the logic was much less straightforward, the design was awkward, and adding/removing additional models/databases would have required rewriting the algorithm. It wasn't a good design and the minimal optimization gains, in my opinion, did not justify the lack of clarity and flexibility.
https://github.com/ic4f/galaxy/blob/7a34c3388feaef05b2d0de73844ddd855743cfcc/lib/galaxy/model/migrations/__init__.py

Usage

See README.md

Testing

1. Unit tests + infrastructure located in test/data/model/migrations.
2. An additional github workflow has been added that runs all the migrations tests on postgresql
   9c96664

Following is a description of the main parts of the testing infrastructure.

conftest.py

Contains the following:

1. Composable "atomic" MetaData objects (implemented as fixture factories), containing one table object.
   Used to compose representations of database state. Example:

@pytest.fixture
def gxy_table1():
    def make_table(metadata):
        return sa.Table('gxy_table1', metadata, sa.Column('id', sa.Integer, primary_key=True))
    return make_table

2. MetaData objects containing one or more tables and representing database state. Example:

@pytest.fixture
def metadata_state3_combined(
    gxy_table1, gxy_table2, tsi_table1, tsi_table2, sqlalchemymigrate_table
):
    metadata = sa.MetaData()
    gxy_table1(metadata)
    gxy_table2(metadata)
    tsi_table1(metadata)
    tsi_table2(metadata)
    sqlalchemymigrate_table(metadata)
    return metadata

The object above is composed of 5 tables defined in fixture factories.

These MetaData objects are used to load a database with a given state. For example, the following test function would load this metadata into the provided database:

def test_state3_combined(engine, metadata_state3_combined):
    with engine.connect() as conn:
        metadata_state3.create_all(bind=conn)

The database states are generated in fixtures (see bottom of test_migrations.py + inline comments for more details).
Each state has 3 versions, distinguished by suffix:

• _gxy (galaxy database that holds gxy* and migration version tables)
• _tsi (tool_shed_install database that holds tsi* and migration version tables)
• _combined (combined database that holds gxy*, tsi*, and migration version tables)

The following states are represented:

• State 1: Non-empty database, no version table.
  (Most ancient state)
• State 2: SQLAlchemy Migrate version table added.
  (Oldest state versioned with SQLAlchemy Migrate)
• State 3: New table added.
  (Last (most recent) state versioned with SQLAlchemy Migrate)
• State 4: Alembic version table added.
  (Oldest state versioned with Alembic)
• State 5: SQLAlchemy Migrate version table removed.
  (Oldest state versioned with Alembic that does not include the SQLAchemy Migrate version table)
• State 6: New table added.
  (Most recent state versioned with Alembic. This is the current state)
• Additional edge case: gxy at state3, tsi has no SQLAlchemy Migrate table.

(State 0 assumes an empty database, so it needs no state fixtures.)

common.py

Contains helpers (context managers and fixtures) for generating database connection URLs, creating and dropping databases, and creating self-disposing engines. Handles both sqlite and postgresql databases. (mysql not supported yet)

versions/

Contains revision scripts used in some of the tests.

test_migrations.py

Contains:

• tests for AlembicManager
• tests for DatabaseStateCache
• tests for DatabaseVerifier (which handles the migration process) in various contexts (different database state and databse setup: see inline comments for details)
• database fixtures loaded with metadata from common.py
• tests for these database fixtures (verifying that they have the expected state)
• various helper fixtures and functions + tests for helper functions

How to test the changes?

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.
• Instructions for manual testing are as follows:

Following is a testing plan I wrote for myself and followed to manually test the system. It doesn't have to be repeated in its entirety.

To create a revision:

1. ./migrate_db.sh revision --head=gxy@head -m "revision message..." (use "gxy" or "tsi")
2. fill-in upgrade/downgrade empty functions in the generated revision script
   (for examples, see test/data/model/migrations/versions/db1)

For upgrades/downgrades: also try: append --sql to do a dry run

To upgrade to head:

1. ./migrate_db.sh upgrade gxy@head
2. verify changes in database

To upgrade/downgrade to a specific revision:

1. ./migrate_db.sh upgrade rev# (for rev# use several starting chars from the revision id)
2. verify changes in database

1. Initial state: no database, separate database setup
Prep: delete old databases, delete old revisions, edit galaxy.yml

1.1. gxy, sqlite database

• gxy-rev1: create table1
• upgrade gxy@head
• gxy-rev2: create table2
• gxy-rev3: create table3
• upgrade to gxy-rev2 (use revision identifier)
• upgrade to gxy-rev3 (use head)
• downgrade to gxy-rev1 (use revision identifier)
• downgrade to gxy-base (use base)
• current: should see nothing
• upgrade: use alembic relative upgrade: gxy@+2
• downgrade: use alembic relative downgrade: gxy@-2

1.2. tsi, sqlite database

• tsi-rev1: create table1
• upgrade heads
• tsi-rev2: create table2
• tsi-rev3: create table3
• upgrade to tsi-rev2 (use +1)
• upgrade to tsi-rev3 (use head)
• downgrade tsi@-1
• downgrade to tsi@base

1.3. repeat steps 1.1, 1.2 with postgresql

2. Initial state: no database, combined databases setup
Prep: delete old databases, delete old revisions, edit galaxy.yml

2.1. gxy, sqlite

• repeat steps from 1.1

2.2. tsi, sqlite

• repeat steps from 2.1

2.3. repeat steps 2.1, 2.2 with postgresql

3. Initial state: old database (create state in another git branch), separate databases setup

3.1. sqlite, run Galaxy:

• no SQLAlchemy Migrate table in gxy: verify fail
• old SQLAlchemy Migrate table in gxy: verify fail
• new SQLAlchemy Migrate table in gxy, no automigrate: verify fail
• new SQLAlchemy Migrate table in both, automigrate: verify upgrade

3.2. postgres, run Galaxy:

• no SQLAlchemy Migrate table in gxy: verify fail
• old SQLAlchemy Migrate table in gxy: verify fail
• new SQLAlchemy Migrate table in gxy, no automigrate: verify fail
• new SQLAlchemy Migrate table in both, automigrate: verify upgrade

4. Initial state: old database (create state in another git branch), separate databases setup, sqlite, gxy is with alembic and current
Run Galaxy:

• no SQLAlchemy Migrate table in tsi: verify fail
• old SQLAlchemy Migrate table in tsi verify fail
• new SQLAlchemy Migrate table in tsi, no automigrate: verify fail

5. Initial state: old database (create state in another git branch), combined database, sqlite
Run Galaxy:

• no SQLAlchemy Migrate table: verify fail
• old SQLAlchemy Migrate table: verify fail
• new SQLAlchemy Migrate table, no automigrate: verify fail
• new SQLAlchemy Migrate table, automigrate: verify upgrade

6. Initial state: old database (create state in another git branch), combined database, postgres
Run Galaxy:

• no SQLAlchemy Migrate table: verify fail
• old SQLAlchemy Migrate table: verify fail
• new SQLAlchemy Migrate table, no automigrate: verify fail
• new SQLAlchemy Migrate table, automigrate: verify upgrade

License

• I agree to license these contributions under Galaxy's current license.
• I agree to allow the Galaxy committers to license these and all my past contributions to the core galaxy codebase under the MIT license. If this condition is an issue, uncheck and just let us know why with an e-mail to [email protected].

[mvdbeek]

I will test this against test.galaxyproject.org and then merge

[jdavcs]

@mvdbeek conflicts resolved.

[mvdbeek]

In its current form this is also a breaking change for the galaxy role:

TASK [galaxyproject.galaxy : Get current Galaxy DB version] **************************************************************************
fatal: [galaxy07.tacc.utexas.edu]: FAILED! => {"changed": false, "cmd": ["/cvmfs/test.galaxyproject.org/venv/bin/python", "/cvmfs/test.galaxyproject.org/galaxy/scripts/manage_db.py", "-c", "/srv/galaxy/test/config/galaxy.yml", "db_version"], "delta": "0:00:03.787073", "end": "2022-01-17 07:38:57.740482", "failed_when_result": true, "msg": "non-zero return code", "rc": 2, "start": "2022-01-17 07:38:53.953409", "stderr": "/cvmfs/test.galaxyproject.org/venv/bin/python: can't open file '/cvmfs/test.galaxyproject.org/galaxy/scripts/manage_db.py': [Errno 2] No such file or directory", "stderr_lines": ["/cvmfs/test.galaxyproject.org/venv/bin/python: can't open file '/cvmfs/test.galaxyproject.org/galaxy/scripts/manage_db.py': [Errno 2] No such file or directory"], "stdout": "", "stdout_lines": []}

obviously this can be adapted, but to merge this into 22.01 I think we'd need some time to prepare and work out the up/downgrade.

[bgruening]

This works nicely in my hand. @ic4f can you please resolve the merge conflicts and then I think we should get this in.

[jdavcs]

@bgruening Thank you! I'll rebase today.

[jdavcs]

This is superseded by #13513. The new branch has a different sequence of commits: (1) some of the old commits have been rearranged and/or squashed, and (2) new commits have been added after a rebase as a manual merge - both as steps in resolving the conflicts above. This branch preserves the original commit sequence, PR description and code review/discussion.