Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DocumentMigrator: support downward migrations #150301

Closed
pgayvallet opened this issue Feb 6, 2023 · 0 comments · Fixed by #153921
Closed

DocumentMigrator: support downward migrations #150301

pgayvallet opened this issue Feb 6, 2023 · 0 comments · Fixed by #153921
Assignees
@pgayvallet
Labels
Epic:ZDTmigrations Zero downtime migrations Feature:Migrations Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc

Comments

@pgayvallet
Copy link
Contributor

pgayvallet commented Feb 6, 2023 •
edited
Loading

Part of #150296
Depends on #150297

The global idea is to be able to have a compatibility layer between the old and new migrations, so that the migrator doesn't need to hold the logic to handle each individual migration source.

  • Combine old/new migrations into unified migration data structure
  • Document migrator to use new data structure
  • Document migrator to support up/down
  • Document migrator to support converting to a specified version (compared to always the last one as currently)
  • model version converted to a virtual model version of an higher major (e.g MV 3 -> 10.3.0)

We also need to figure out what to do with the coreMigrationVersion. This will need to switch to decoupled versioning too, and we need to think about when we need to downward migration core version (likely only rollbacks). This part could be done as a follow-up

PRs:
@pgayvallet pgayvallet added Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc Feature:Migrations Epic:ZDTmigrations Zero downtime migrations labels Feb 6, 2023
This was referenced Feb 6, 2023
Closed
Closed
Merged
@pgayvallet pgayvallet self-assigned this Feb 8, 2023
@pgayvallet pgayvallet mentioned this issue Feb 8, 2023
Merged
pgayvallet added a commit that referenced this issue Feb 8, 2023
@pgayvallet
Preliminary DocumentMigrator cleanup (#150529)
91ff710 
Related to #150301

This PR tend to do some preliminary cleanup before diving into the
actual implementation. I'm mostly extracting things from the (way too
big and hard to read) document migrator, to increase readability,
maintainability and testability (note: I did not add additional unit
test for the extracted files, as I did not change the impl and as we'll
be rewriting them all soon enough - it would just have been a loss of
time)

- Move the document migrator from
`packages/core/saved-objects/core-saved-objects-migration-server-internal/src/core`
to it's dedicated folder `document_migrator` folder
- Extract all the bits not directly performing the document
transformation (e.g building the migration map, validation, utils) to
their dedicated files
- Only moving things around. No concrete code (apart from types) was
modified, to ease managing conflicts with
#150443
@pgayvallet pgayvallet mentioned this issue Feb 10, 2023
Merged
pgayvallet added a commit that referenced this issue Feb 15, 2023
@pgayvallet @kibanamachine
register model version migration to the document migrator (#150842)
e64d545 
## Summary

Part of #150301

- Add logic to converts model version transformations to the format used
by the document migrator
- Use in when preparing migration data for the document migrator
- Improve the migration validation logic to take model versions into
account (and clean it)

---------

Co-authored-by: kibanamachine <[email protected]>
justinkambic pushed a commit to justinkambic/kibana that referenced this issue Feb 23, 2023
@pgayvallet @kibanamachine @justinkambic
register model version migration to the document migrator (elastic#15…
dc8ab4d 
…0842)

## Summary

Part of elastic#150301

- Add logic to converts model version transformations to the format used
by the document migrator
- Use in when preparing migration data for the document migrator
- Improve the migration validation logic to take model versions into
account (and clean it)

---------

Co-authored-by: kibanamachine <[email protected]>
@pgayvallet pgayvallet mentioned this issue Mar 29, 2023
Merged
pgayvallet added a commit that referenced this issue Apr 3, 2023
@pgayvallet @kibanamachine
DocumentMigration: implement downward transformation of documents (#1…
54df909 
…53921)

## Summary

Fix #150301

Implements `DocumentMigrator.transformDown` and the associated
transformation pipeline.

Note: The feature is currently unused, and will only be when we'll start
#150312

---------

Co-authored-by: kibanamachine <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pgayvallet pgayvallet

Labels
Epic:ZDTmigrations Zero downtime migrations Feature:Migrations Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

DocumentMigration: implement downward transformation of documents pgayvallet/kibana
1 participant
@pgayvallet