Recreate all Django migrations

procrastinate/contrib/django/README.md

@@ -4,101 +4,14 @@
 
 See https://procrastinate.readthedocs.io/en/stable/howto/django.html
 
-## Contributing
+## Contributing: Migrations
 
-### How are the migrations dynamically generated
+Whenever a migration is added to procrastinate/sql/migrations, you should
+add a corresponding Django migration in procrastinate/contrib/django/migrations.
 
-In the following `procrastinate.contrib.django.migrations` is noted `p.c.d.migrations`
-for conciseness.
+For now, this is a manual process. Look at existing migrations and copy one
+of them, you'll need to change:
 
-The end goal is that when we run `./manage.py migrate`, Django sees one Django migration
-for each sql migration in our `procrastinate/sql/migrations` folder. This means we want
-to "trick" Django into thinking that there are migrations in the `p.c.d.migrations`
-package, but this package doesn't exist on the disk.
-
-This all happens in  `migrations_magic` if you want to read the code along.
-
-Django introspects its apps looking for migrations (in django.db.migrations.loader). It
-doesn't specifically let each App override the way it's done, so we can't "just"
-override the right method in App and be done with it. But Django doesn't read the
-migrations modules on the disk directly either. It does the following:
-
-A. Try to import `p.c.d.migrations` using `importlib.import_module`. If it doesn't
-  work, consider the app doesn't provide migrations
-B. Use `pkgutil.iter_modules()` to discover all modules under `p.c.d.migrations`
-C. For each module, import it using using `importlib.import_module` and read its
-  `Migration` class
-
-Python's import mechanisms are modular, there's multiple ways we can add some code for
-our use case. While not necessary for the following, reading Python's import
-documentation will help: https://docs.python.org/3/reference/import.html
-
-We could imagine doing step A by going either way: either `p.c.d.migrations` could exist
-and be empty or we could trick Python's import system into believing it exists. The
-first alternative seems simpler, but if we do that, in the B step,
-`pkgutil.iter_modules` will try to load submodules the same way this module was loaded:
-by reading the disk. So in the end, given we'll be tricking the import system anyway,
-we're doing it for both `p.c.d.migrations` and `p.c.d.migrations.*`
-
-Doing step A is done the following way: we ensure that before Django tries to load
-migrations, this module's `load()` function is called. It adds an instance of
-`ProcrastinateMigrationsImporter` at the end of the `sys.meta_path`. This means that
-whenever a module will be imported and not be found on the disk, our instance's
-`find_spec()` method will be called. This method should return a `ModuleSpec` instance
-if we want the finder to take responsibility on loading the module, None otherwise. We
-return a ModuleSpec when the path we want to load is `p.c.d.migrations`, and provide a
-loader instance, that will be used for loading our virtual module. Our importer class
-serves as both finder and loader, so loader is self.
-
-Python will create an empty module for us with `__name__` set to the dotted path of the
-module we want to run, and call the loader's exec_module method, where our role is to
-put the content of the module in place. In particular, the module's `__path__` is
-important. A module is a package (i.e. it has submodules) if it has a `__path__`. Also,
-the `__path__` will be used for `pkgutil.iter_modules`. At this step, we could put a
-plausible value in the `__path__` like, but there's no real file on the disk we can
-match, and this would require us to figure out where we are on the disk. A better
-solution is to use a value that we know won't be mistaken for a path, and that we're
-sure we can recognize easily. That's what we do with `VIRTUAL_PATH`.
-
-Step B: Ok, we've returned a module for `p.c.d.migrations`, now Django will be running
-`pkgutil.iter_modules` on our module's `__path__` (`VIRTUAL_PATH`). This results in
-Python calling all the callables in `sys.path_hooks` with our path until one doesn't raise
-`ImportError`. The default path hooks will all fail, we need our own hook. That's why in
-the `load()` function mentionned below, we also added our importer's `path_hook` method to
-`sys.path_hook`. We get called and check that the path is the `VIRTUAL_PATH` that we set
-earlier. The path hook needs to return a Finder object suitable for listing its
-submodules, so that would be `self`
-
-Now Python will call a non-standard method on our finder: `iter_modules`
-(https://docs.python.org/3/library/pkgutil.html?highlight=iter_modules#`pkgutil.iter_modules`)
-This method is expected to return tuples consisting of a submodule name and a boolean
-indicating whether the submodule is itself a package or not (easy: in our case it's
-not). We'll expand below on how we got ahold of the fake migrations modules we want
-to expose, but for now, let's assume we have them. We're able to return a list of
-module names.
-
-Step C: Now, Django will iterate on our module names and call `importlib.import_module`.
-It works the same as before: Python will not manage to load the modules on disk, so
-following the order of `sys.meta_path`, it will ask our importer's `find_spec()`. This
-method returns `ModuleSpec` objects both for `p.c.d.migrations` and anything below, and
-indicates itself to be the loader. This means the `exec_module` method is called next,
-and this time, we can attach the `Migration` class that we have prepared to the module,
-based on the module's `__name__` attribute.
-
-Django will be happy with that and consider our app to have valid migrations.
-
-But how did we generate the proper `Migration` classes? This part is actually much
-more readable:
-
-- We use `importlib_resources.files` to get the path of each sql file in
-  procrastinate's migration folder (ensuring this would work even if our package was
-  actually in a zip)
-- For each migration, we build a `ProcrastinateMigration` object that extracts various
-  parts from the migration filename
-- For each `ProcrastinateMigration`, we generate a new class inheriting Django's
-  `Migration` class. It needs to know the migration operations, its name and index
-  (`0001`, `0002`, ...) and be linked to the previous migration
-- The migration operations consist of a single step: a RunSQL operation whose SQL
-  is the contents of the corresponding procrastinate migration file (read with `importlib_resources`)
-
-That's it.
+-   the reference to the SQL migration file name
+-   the name of the django migration
+-   the name of the previous migration

procrastinate/contrib/django/migrations/0001_initial.py

@@ -0,0 +1,11 @@
+from __future__ import annotations
+
+from django.db import migrations
+
+from .. import migrations_utils
+
+
+class Migration(migrations.Migration):
+    initial = True
+    operations = [migrations_utils.RunProcrastinateSQL(name="00.00.00_01_initial.sql")]
+    name = "0001_initial"

procrastinate/contrib/django/migrations/0002_drop_started_at_column.py

@@ -0,0 +1,15 @@
+from __future__ import annotations
+
+from django.db import migrations
+
+from .. import migrations_utils
+
+
+class Migration(migrations.Migration):
+    operations = [
+        migrations_utils.RunProcrastinateSQL(
+            name="00.05.00_02_drop_started_at_column.sql"
+        )
+    ]
+    name = "0002_drop_started_at_column"
+    dependencies = [("procrastinate", "0001_initial")]

procrastinate/contrib/django/migrations/0003_drop_started_at_column.py

@@ -0,0 +1,15 @@
+from __future__ import annotations
+
+from django.db import migrations
+
+from .. import migrations_utils
+
+
+class Migration(migrations.Migration):
+    operations = [
+        migrations_utils.RunProcrastinateSQL(
+            name="00.05.00_01_drop_started_at_column.sql"
+        )
+    ]
+    name = "0003_drop_started_at_column"
+    dependencies = [("procrastinate", "0002_drop_started_at_column")]

procrastinate/contrib/django/migrations/0004_drop_procrastinate_version_table.py

@@ -0,0 +1,15 @@
+from __future__ import annotations
+
+from django.db import migrations
+
+from .. import migrations_utils
+
+
+class Migration(migrations.Migration):
+    operations = [
+        migrations_utils.RunProcrastinateSQL(
+            name="00.05.00_03_drop_procrastinate_version_table.sql"
+        )
+    ]
+    name = "0004_drop_procrastinate_version_table"
+    dependencies = [("procrastinate", "0003_drop_started_at_column")]

procrastinate/contrib/django/migrations/0005_fix_procrastinate_fetch_job.py

@@ -0,0 +1,15 @@
+from __future__ import annotations
+
+from django.db import migrations
+
+from .. import migrations_utils
+
+
+class Migration(migrations.Migration):
+    operations = [
+        migrations_utils.RunProcrastinateSQL(
+            name="00.06.00_01_fix_procrastinate_fetch_job.sql"
+        )
+    ]
+    name = "0005_fix_procrastinate_fetch_job"
+    dependencies = [("procrastinate", "0004_drop_procrastinate_version_table")]

procrastinate/contrib/django/migrations/0006_fix_trigger_status_events_insert.py

@@ -0,0 +1,15 @@
+from __future__ import annotations
+
+from django.db import migrations
+
+from .. import migrations_utils
+
+
+class Migration(migrations.Migration):
+    operations = [
+        migrations_utils.RunProcrastinateSQL(
+            name="00.07.01_01_fix_trigger_status_events_insert.sql"
+        )
+    ]
+    name = "0006_fix_trigger_status_events_insert"
+    dependencies = [("procrastinate", "0005_fix_procrastinate_fetch_job")]

procrastinate/contrib/django/migrations/0007_add_queueing_lock_column.py

@@ -0,0 +1,15 @@
+from __future__ import annotations
+
+from django.db import migrations
+
+from .. import migrations_utils
+
+
+class Migration(migrations.Migration):
+    operations = [
+        migrations_utils.RunProcrastinateSQL(
+            name="00.08.01_01_add_queueing_lock_column.sql"
+        )
+    ]
+    name = "0007_add_queueing_lock_column"
+    dependencies = [("procrastinate", "0006_fix_trigger_status_events_insert")]

procrastinate/contrib/django/migrations/0008_close_fetch_job_race_condition.py

@@ -0,0 +1,15 @@
+from __future__ import annotations
+
+from django.db import migrations
+
+from .. import migrations_utils
+
+
+class Migration(migrations.Migration):
+    operations = [
+        migrations_utils.RunProcrastinateSQL(
+            name="00.10.00_01_close_fetch_job_race_condition.sql"
+        )
+    ]
+    name = "0008_close_fetch_job_race_condition"
+    dependencies = [("procrastinate", "0007_add_queueing_lock_column")]

procrastinate/contrib/django/migrations/0009_add_defer_job_function.py

@@ -0,0 +1,15 @@
+from __future__ import annotations
+
+from django.db import migrations
+
+from .. import migrations_utils
+
+
+class Migration(migrations.Migration):
+    operations = [
+        migrations_utils.RunProcrastinateSQL(
+            name="00.10.00_02_add_defer_job_function.sql"
+        )
+    ]
+    name = "0009_add_defer_job_function"
+    dependencies = [("procrastinate", "0008_close_fetch_job_race_condition")]

procrastinate/contrib/django/migrations/0010_add_procrastinate_periodic_defers.py

@@ -0,0 +1,15 @@
+from __future__ import annotations
+
+from django.db import migrations
+
+from .. import migrations_utils
+
+
+class Migration(migrations.Migration):
+    operations = [
+        migrations_utils.RunProcrastinateSQL(
+            name="00.11.00_03_add_procrastinate_periodic_defers.sql"
+        )
+    ]
+    name = "0010_add_procrastinate_periodic_defers"
+    dependencies = [("procrastinate", "0009_add_defer_job_function")]

procrastinate/contrib/django/migrations/0011_add_foreign_key_index.py

@@ -0,0 +1,15 @@
+from __future__ import annotations
+
+from django.db import migrations
+
+from .. import migrations_utils
+
+
+class Migration(migrations.Migration):
+    operations = [
+        migrations_utils.RunProcrastinateSQL(
+            name="00.12.00_01_add_foreign_key_index.sql"
+        )
+    ]
+    name = "0011_add_foreign_key_index"
+    dependencies = [("procrastinate", "0010_add_procrastinate_periodic_defers")]

procrastinate/contrib/django/migrations/0012_add_locks_to_periodic_defer.py

@@ -0,0 +1,15 @@
+from __future__ import annotations
+
+from django.db import migrations
+
+from .. import migrations_utils
+
+
+class Migration(migrations.Migration):
+    operations = [
+        migrations_utils.RunProcrastinateSQL(
+            name="00.14.00_01_add_locks_to_periodic_defer.sql"
+        )
+    ]
+    name = "0012_add_locks_to_periodic_defer"
+    dependencies = [("procrastinate", "0011_add_foreign_key_index")]

procrastinate/contrib/django/migrations/0013_fix_procrastinate_defer_periodic_job.py

@@ -0,0 +1,15 @@
+from __future__ import annotations
+
+from django.db import migrations
+
+from .. import migrations_utils
+
+
+class Migration(migrations.Migration):
+    operations = [
+        migrations_utils.RunProcrastinateSQL(
+            name="00.15.02_01_fix_procrastinate_defer_periodic_job.sql"
+        )
+    ]
+    name = "0013_fix_procrastinate_defer_periodic_job"
+    dependencies = [("procrastinate", "0012_add_locks_to_periodic_defer")]

procrastinate/contrib/django/migrations/0014_add_finish_job_and_retry_job_functions.py

@@ -0,0 +1,15 @@
+from __future__ import annotations
+
+from django.db import migrations
+
+from .. import migrations_utils
+
+
+class Migration(migrations.Migration):
+    operations = [
+        migrations_utils.RunProcrastinateSQL(
+            name="00.16.00_01_add_finish_job_and_retry_job_functions.sql"
+        )
+    ]
+    name = "0014_add_finish_job_and_retry_job_functions"
+    dependencies = [("procrastinate", "0013_fix_procrastinate_defer_periodic_job")]

procrastinate/contrib/django/migrations/0015_add_trigger_on_job_deletion.py

@@ -0,0 +1,15 @@
+from __future__ import annotations
+
+from django.db import migrations
+
+from .. import migrations_utils
+
+
+class Migration(migrations.Migration):
+    operations = [
+        migrations_utils.RunProcrastinateSQL(
+            name="00.17.00_01_add_trigger_on_job_deletion.sql"
+        )
+    ]
+    name = "0015_add_trigger_on_job_deletion"
+    dependencies = [("procrastinate", "0014_add_finish_job_and_retry_job_functions")]

procrastinate/contrib/django/migrations/0016_delete_finished_jobs.py

@@ -0,0 +1,15 @@
+from __future__ import annotations
+
+from django.db import migrations
+
+from .. import migrations_utils
+
+
+class Migration(migrations.Migration):
+    operations = [
+        migrations_utils.RunProcrastinateSQL(
+            name="00.17.00_02_delete_finished_jobs.sql"
+        )
+    ]
+    name = "0016_delete_finished_jobs"
+    dependencies = [("procrastinate", "0015_add_trigger_on_job_deletion")]

procrastinate/contrib/django/migrations/0017_add_checks_to_finish_job.py

@@ -0,0 +1,15 @@
+from __future__ import annotations
+
+from django.db import migrations
+
+from .. import migrations_utils
+
+
+class Migration(migrations.Migration):
+    operations = [
+        migrations_utils.RunProcrastinateSQL(
+            name="00.17.00_03_add_checks_to_finish_job.sql"
+        )
+    ]
+    name = "0017_add_checks_to_finish_job"
+    dependencies = [("procrastinate", "0016_delete_finished_jobs")]

procrastinate/contrib/django/migrations/0018_add_checks_to_retry_job.py

@@ -0,0 +1,15 @@
+from __future__ import annotations
+
+from django.db import migrations
+
+from .. import migrations_utils
+
+
+class Migration(migrations.Migration):
+    operations = [
+        migrations_utils.RunProcrastinateSQL(
+            name="00.17.00_04_add_checks_to_retry_job.sql"
+        )
+    ]
+    name = "0018_add_checks_to_retry_job"
+    dependencies = [("procrastinate", "0017_add_checks_to_finish_job")]