Add support of directory structure for running a migration

[sebastienbeau]

Hi All

context

We face to some limitation, difficulty with marabunta mostly it's the following point:

• setting a version in a PR is needed to run the migration script but it's a none sense because we do not know yet the version
• when merging multiple PR we have conflict

Proposal

The main point will to allow directory structure for migration instead of yaml file

Something like

+ migration
  + 1.0.0
     - pre-script-bar
     - pre-script-foo
     - post-script-1
     - addons-bar
     - addons-foo
  + 1.0.1
     - ...
  + 1.1.0
     - ...
  + current
     - ... 

pre-script-* and post-script-* can be whatever script you want (bash exec with anthem cmd, python script based on click-odoo)

addons-* are text file with list of module, they will use to get the list of module to update (and remove duplicate module)

Process:

When you do a PR you add you script (pre, post) and the addons list to update inside the current directory.
Every PR will do the same so no conflict are possible (just name your script correctly)

When you want to do release you just need to git move all file inside a new directory with the name of the release.

Implementation

I see two possibility to achieve my goal, doing a custom script that will generate the migration script

Or adding two option into marabunta

--migration-directory
MARABUNTA_MIGRATION_DIRECTORY

--migration-directory-process-current
MARABUNTA_MIGRATION_DIRECTORY_PROCESS_CURRENT

That will read the information of the migration-file and the directory information to perform the upgrade.

I do not think it worth to add an extra layer of directory for MARABUNTA_MODE (at least in our case we do not need it). The idea is to only set the MARABUNTA_MIGRATION_DIRECTORY with real database (prod and CI) and only MARABUNTA_MIGRATION_DIRECTORY_PROCESS_CURRENT for CI for testing the deploy.

What do you think ? Inside or outside marabunta ?

[sebastienbeau]

@guewen @simahawk

[guewen]

Hi, thanks for opening the discussion!

On the concept, sure it makes sense.

What would be --migration-directory-process-current for?

[guewen]

poke @p-tombez

[sebastienbeau]

It's for running the migration script that are inside "current" directory.

The idea is to only run them in CI, never in production. Normally you should never have stuff inside in production as the release have moved the file.

After thinking twice maybe the best is to not have this option and simply add a check when running marabunta in prod mode and raise an error if there is something inside the current directory

[simahawk]

I was thinking about a similar change some time ago.
I think that it would be easier to simply generate the mig file automatically if you find a migrations.d folder w/ proper content in it. This way you won't have to hack that much how it runs right now.

In any case, thinking out loud, this thing about "pre-*" "post-" reminds me of how odoo standard migr machinery works in a module and then I wonder: aren't we reinventing the wheel?
The only missing point here would be to support an undefined "current" release.

[sebastienbeau]

Yes I am asking the same thing about pre-* post-*.
Should be use Odoo module upgrade instead? I known that Acsone is using this.
The good point of using odoo upgrade is to make every dev familiar with it (easier to contribute to openupgrade, easier to create migration script for OCA module)

The only difference is that this pre- can be play with marabunta before updating any module.

@guewen what is your point of view on this?

[sebastienbeau]

Note : for running upgrade Acsone is using click-odoo-update and update all module that have changed by default and put all migration/config script into odoo module

[guewen]

Another good point of using odoo upgrades is if we have to deploy a project on odoo.sh, we'd use the same method.

On the downsides, sometimes we couldn't even start odoo without running a pre script fixing something using SQL first (IIRC datamodel change on res_users), so we couldn't run odoo upgrade scripts. But maybe that's fixed now?
And generally I guess we have more control with the way marabunta is using but maybe it's not really required?

[sebastienbeau]

On the downsides, sometimes we couldn't even start odoo without running a pre script fixing something using SQL first (IIRC datamodel change on res_users), so we couldn't run odoo upgrade scripts. But maybe that's fixed now?

I was asking myself the same question ! Maybe I should try to only use odoo migration script for testing, and see if I have blocking point.

Maybe we can found some work-around if we really need to run something before running click-odo-update (like adding a temporary entrypoint with sql or python script if need).

I am going to do more test with Odoo native upgrade.