Add allowUnorderedMigrations option to Migrator

[tracehelms]

What is this?

This add an allowUnorderedMigrations option to MigratorProps. The options are true or false, with false being the default.

When true, it will allow new migrations to be run even if they are added alphabetically before ones that have already executed.

When false, it will give an error if your migration files are not alphabetically ordered in the same order that they were executed in the database.

Closes #697

Why?

If a team has multiple developers, two people can create migrations asynchronously on separate branches. With strict ordering, those migration files have to be merged in the right order (or renamed before merging) or else Kysely will throw an error when trying to migrate them.

Example:

- Developer A creates migration `2023-09-13-migration-A.ts`
- Developer B creates migration `2023-09-14-migration-B.ts`
- Both developers make a PR at roughly the same time
- Developer B merges their PR and deploys
- Developer A merges their PR
- Developer A deploys and tries to run migrations

Kysely will give an error for this with strict ordering. With permissive ordering, the unrun migration is executed.

Pending migrations vs. executed migrations

This also adds the concept of pending (unexecuted) migrations to Migrator. When migrating up, Migrator will run pending migrations in order for n steps. When migrating down, Migrator will consult the database for executed migrations and travel down that list n steps.

This doesn't change the current functionality for ensuring ordered migrations, but also allows out of order ones.

Some examples:
(* denotes current migration version)

New migration:

Migrations Already Ran:  0  1  2*
Migration Files:         0  1  2  3

Migrate up:              0  1  2  3*

Migrate down one:

Migrations Already Ran:  0  1  2*
Migration Files:         0  1  2  3

Migrate down:            0  1*

New out-of-order migration:

Migrations Already Ran:        0  1  2     4*
Migration Files:               0  1  2  3  4  5

Migrate up (strict):           error
Migrate up (permissive):       0  1  2  3  4*
Migrate up again (permissive): 0  1  2  3  4  5*

Migrate down with out-of-order migration:

Migrations Already Ran:    0  1  2     4*
Migration Files:           0  1  2  3  4  5

Migrate down (strict):     error
Migrate down (permissive): 0  1  2*

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
kysely	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jan 2, 2024 5:53pm

[igalklebanov]

Hey 👋

Thanks! 💪

I feel like allowUnorderedMigrations seems clearer, or am I missing something?

[tracehelms]

I feel like allowUnorderedMigrations seems clearer, or am I missing something?

That would definitely be clearer, good thinking. I'll update and post up the changes.

[tracehelms]

@igalklebanov Done, and I updated the docs / comments too 👍

[tracehelms]

Hi @igalklebanov and @koskimas, I was just wondering if you've gotten a chance to look at this and what you think. Let me know if you've got any questions / requests 😄 Thanks!

[WyattMcMurry]

I think this is a really necessary change for coordinating across multiple teams, environments, and timezones.
Edit: just wanted to add that Kyesely rocks!

[danolife]

Yes can you please release this already 😭
My production is broken because of unordered migrations, I don't want to update everything manually again!
Please @igalklebanov @koskimas 🙏

[tracehelms]

This branch was out of date with master so I rebased and pushed up the changes 👍

[blbrown13]

This would be super helpful for our team's workflow! 😍 Hopefully it gets approved and merged soon (cc @igalklebanov @koskimas ) 🙏🏻

[avocadojesus]

This would be really useful for my team as well, coordinating migrations is very tricky for us and this would solve all of our problems. Other than this little snag, Kysely has been absolutely incredible to use, thanks so much to the maintainers for writing such an incredible library!!!

[koskimas]

Now that I have a week off from work, I finally have some time for Kysely and could review this one.

@danolife We're not keeping these PRs waiting just to be mean or because we hate your team. We simply haven't had time.

@tracehelms This seems excellent and will definitely get merged! Could you rebase this one more time?

[tracehelms]

Hey @koskimas, thanks for reviewing! I'll get this rebased and patched up with your changes. 👍

[tracehelms]

Ready for another look @koskimas 👍

[koskimas]

Perfect!

[tracehelms]

Awesome, thanks for reviewing and merging! 💯

[koskimas]

This is now released in 0.27.2

[loremru]

Actually allowUnorderedMigrations doesn't work

[igalklebanov]

@loremru this PR contains a unit test that proves it does. Thus makes me highly doubt your error has anything to do with this feature.

[loremru]

@igalklebanov
I think #resolveMigrations should not sort my migrations if I set allowUnorderedMigrations to true.
When I removed this line, error disappears

[igalklebanov]

@loremru you're misunderstanding the feature. It is about not throwing migrator-specific errors if a new migration file is added that is in the middle of the sorted ran migrations.

e.g.
PR 1 has 0003-a.js
PR 2 has 0003-b.js

If PR 2 gets merged first, running the migrations once PR 1 is merged would fail because:

[x] 0001
[x] 0002
[ ] 0003-a
[x] 0003-b

This feature makes it not fail.

In your case, the migrations are poorly named. You should at least pad left with 0s. This probably results in sql errors, not migrator errors as the historic order is wrong between migrations that access the same resource. e.g. alter before create.