Use aerich for database migrations

[bigherc18]

This PR is about using aerich to add support for database migrations

Here's a demo: https://asciinema.org/a/Sw7OwQZKmWyznV3xgohjd59OT

I added a new config called migrations_dir to specify where migration files should be stored

database:
  migrations_dir: ./migrations

A a set of commands have been added to manage database migrations:

• dipdup schema history to show the migration history
• dipdup schema heads to show the show the current available heads
• dipdup schema migrate to create a new migration changes file
• dipdup schema upgrade to upgrade the database to the latest or a specified version
• dipdup schema downgrade to downgrade the database to a previous version

Theses commands invoke aerich's commands under the hood

The migrations directory should be initialized before using any of these commands, either with dipdup schema init or by running the indexer with dipdup run.

Note that aerich commands can't be used directly, as it expects the config to be stored in pyproject.toml

[droserasprout]

Hi! Thanks for your contribution! It's a great code with many smart solutions; I like every piece of it! My only concern is making this integration optional in every way: dependencies, imports, files in the project package.

[bigherc18]

Do you guys prefer rebase or merge with next to fix the conflicts ?

[droserasprout]

@bigherc18 We squash every merge request, so it's up to you what's inside.

[bigherc18]

Some concerns and limitations I have:

• aerich will be confused if the user changes the database. If we want to support changing the database, we have to make aerich aware of which database is being used.
• If you answer "No" to the question "Downgrade is dangerous, which may cause data loss. Are you sure? [y/N]" after running dipdup schema downgrade, the CLI hangs there forever.
• aerich has confusing messages after running heads and history. It makes the user feel like they have to take action when everything is actually fine: "No available heads, try migrate first."

[droserasprout]

Your last comment shows such a great amount of testing done! 🥰 I think, there's nothing critical. Yes, running commands in random order on multiple databases sometimes confuse aerich, but all real cases works perfectly.

I have just a few nitpicks. Then make sure that make all docs command passes, add yourself to the thanks.md file, and let's merge it!

[bigherc18]

I think, there's nothing critical. Yes, running commands in random order on multiple databases sometimes confuse aerich, but all real cases works perfectly.

Thank you for your feedback but I just want to make sure I explained my concerns clear enough for you

1. Multiple databases not supported by migrations

TLDR; if a user uses multiple databases at the same time, migrations is not supported

I see dipdup makes it easy for users to swap databases by just combining different config files with multiple -c args, the problem is: if a user swaps his database after doing some migrations in the first one, aerich is now really confused and I don't even know what will happen, the aerich table will be empty (or contains migrations history for the second database) but the migrations folder still contains files from the first database

I think this worth at least a note in the docs

3. CLI handing

Here's a little video demonstrating the issue: