Create an utility for IPFS Repo migrations

[pgte]

Repo migrations should, if possible, happen automatically.
This would minimize disruption on projects that are using long-running js-ipfs repos, like peerpad.

[daviddias]

I don't necessarily agree with the automatically part since that a bad repo migration might lead to data corruption. Of course, we will always strive to make it correct, but the user should be informed and make the decision.

This wasn't an issue before because there is a tool just for that -- https://github.com/ipfs/fs-repo-migrations --, but know that we have long-running browser applications such as PeerPad, we do need one in JavaScript as well 👍

[pgte]

A separate migration tool can be created, but we'll end up with these apps always including this if they want to be able to survive repo upgrades..
It should be seamless to the user anyway.
I'd prefer it being seamless for the app developer too..
Data corruption is always the responsibility of the library doing the migration, whether you choose to include that in js-ipfs or not.. :)

[mkg20001]

A good idea would be auto-migration and adding an option such as "disableAutoMigration" that would make IPFS throw an error if it needs to migrate such that the application could ask the user to backup the data first, etc.

[daviddias]

I was just coming here to also give the flag idea :) thanks @mkg20001 :)

[daviddias]

@pgte solution for PeerPad https://github.com/ipfs-shipyard/peerpad-core/blob/master/src/backend/migrate-ipfs-repo-if-necessary.js

[olizilla]

I'm a big +1 on this. We're working on adding js-ipfs to ipfs-companion and we're going to want to upgrade js-ipfs regularly. Any app developer that uses js-ipfs as a dep is going to need repo migration for existing users. I'd rather ask js-ipfs to do that migration than have to hand roll it with less context at the app level.

[AuHau]

Hey there,
If you would be ok with it, I would like to work on this. I did some work on go-ipfs and the already mentioned migration utility, so I have some understanding of how it works in go-land.

My proposal is to create a separate package (for example "js-ipfs-repo-migrator"?) which will hold all the migrations and ability to execute them. These are the following features/requirements I thought of:

• Migration's interface - the tool will define an interface which all the migrations will have to follow.
• Reversibility - an ability to revert to previous versions of the repo (if the migrations allow it).
• API for automatic migrations on jsipfs's bootstrap.
• Basic CLI commands for manual migrations in Node environments.

Let me know if there are more requirements from your side and if I can start working on it.

[AuHau]

Just noticed that @achingbrain is assigned to this. What is the status of it @achingbrain ? Are you working on it?

[AuHau]

Any update? Go/no-go please?

[AuHau]

I am working on this here https://github.com/AuHau/js-ipfs-repo-migrations

Hopefully next week there will be a first version to review.

[alanshaw]

@AuHau ❤️ thanks for picking this up! Apologies for the lack of responses - I somehow missed the notifications on this thread. We had this on our list for Q1 2019 but so far no progress.

I think your list of requirements is great and I'm glad you have knowledge of the go-ipfs repo migration implementation as I was going to suggest that where possible it should have the same or similar functionality. In addition to your requirements I'd add:

• It must work in the browser as well as Node.js
  • It must not add a significant overhead to the bundle size
• There must be a way to programmatically run migrations (for when 'disableAutoMigration' option is used)
• The tool must have good test coverage
  • We must be able to add tests for individual migrations to ensure they succeed as expected
  • We must run tests for the tool and migrations on the browser as well
• There must be adequate documentation on the README
  • There must be instructions on:
    • How to add a new migration
    • How to add a new migration test
    • How to disable automatic repo migrations (programmatically and via command line)
    • How to use the command line utility to run migrations
    • How to programmatically run migrations
  • Some of these instructions should be propogated up to the js-ipfs README
  • There should be other information on the README to conform with standard README spec

Requirements I need to double check on but if you have opinions I'm interested to hear:

• The tool should be able to migrate a Go IPFS repo (for interop)
• Go IPFS repo migrations should be backported to this tool (I'm 👎 on this one!)

Can I also request that when you add code to js-ipfs-repo-migrations that you send a PR and request myself and @achingbrain as reviewers? That way we can deal with any issues as they arrive!

[AuHau]

Yey, thanks for the reply! ❤️

Most of the additional requirements are already incorporated or are planned, so I think we are in sync here! There are few questions&comments from my side.

Currently, how I have designed it is that the migration will receive IPFSRepo() instance which will automatically handle the browser/node environment difference. Of course, there might be more complex migrations that the IPFSRepo or the underlying datastore won't be able to handle, but then the migration's author can always detect what environment the repo lives in using the IPFSRepo options and do more complex operations with native functions. Do you think this is a sufficient mechanism for supporting the migration framework?

Regarding the bundle size, definitely, agree. There is one somehow related thing. The go-ipfs migration tool requires to have all dependencies bundled with the migration to ensure functional migrations in the future. I can completely understand the point of view and I have laid out a similar concept for the JS version, but this methodology might increase the bundle size quite significantly (well depends a bit on point of view). Currently, every migration has its own package.json and the node_modules from the migration is committed into the repo. What do you think about this?

+1 on tests and documentation, that will be definitely part of it!

Regarding the ability to migrate go-ipfs repo, I feel that the js-repo and go-repo are still quite diverse for supporting this (I don't have so deep understanding of the differences, but for example, the key management is very different in go-land from js-land). So I would not aim for that. I understand this tool as mean how to get to the point where the full interop will happen...

From interop point of view, the versioning of the migrations should be same in both lands. But I feel like this will require some cooperation from the go-team as well, because I can imagine that there might be a point where there is something to migrate in js-land but nothing really to migrate in go-land or vice versa. Should there be then support for "empty migrations" which will just bump the version up, to match the other's land versioning?

Lastly, I planned to create the PR when it is ready for review, is that alright or do you want me to create right away so you can track the advance? Also if you would like I can transfer the GitHub repo under IPFS org...

PS.: I am developing it already against the async branches of the dependencies.

[AuHau]

Hmm now when I am thinking bit more about passing the IPFSRepo() instance to the migration, I am not so sure if it is a good idea. Because each migration will expect some specific behavior/interface of the IPFSRepo, but as js-ipfs-repo will be developed further this might change and then it is a question how the change should be propagated to this tool. I see several options:

1. Freeze version of js-ipfs-repo for each migration ==> 👎 bundle size explosion, complexity etc.
2. Have the latest version of js-ipfs-repo as a package dependency and depends on the test coverage to ensure functionality over the js-ipfs-repo updates
3. Drop js-ipfs-repo dependency and don't use any abstraction for the migrations

In some way, I like the simplicity of 2nd option, but it bit defers another rule from go-migration tool, that makes sense: "freeze migrations after publishing them" as it might happen that with js-ipfs-repo updates the (breaking) changes will have to be propagated to the old migrations...

Any ideas?

[AuHau]

Well after some more work and thinking, I can reply to the previous question - having a dependency on js-ipfs-repo does not seem like a good idea and it does not make really sense. New question is, can the migrations depends on datastore interface? Eq. the migration would get either datastore-fs or datastore-level instance of the root of the repo folder?

[AuHau]

Hey there!
Sooo first draft is available for the feedback! It is still not complete, but it is a meaningful unit that can be feedbacked.

You can find the PR here: ipfs/js-ipfs-repo-migrations#1

@alanshaw @achingbrain I have invited you to the repo so I could add you as reviewers.

[AuHau]

Also to see example of how migration will look like see: ipfs/js-ipfs-repo-migrations#2

[AuHau]

@alanshaw @achingbrain so all PRs are ready for review:

#2044
ipfs/js-ipfs-repo-migrations#1

Fyi, I will be offline big part of June, so I can work on your remarks before cca June 10th or then in July.

[daviddias]

@AuHau gave you write access so that in the future you can do PRs from branches within the repo :)

[AuHau]

The current state of this small endeavor:

• js-ipfs-repo-migrations
  • main framework / tests / documentation: Initial work js-ipfs-repo-migrations#1
  • first migration: Migration 8: Encoding key's names to base32 js-ipfs-repo-migrations#2
• integration PRs for js-ipfs-repo-migrations
  • Automatic repo migrations js-ipfs-repo#202
  • [WIP] feat: integration of tool for repo migration #2044
• migration's related work
  • encoded key's names: feat: encoding key's name to base32 #2410
  • blockstore use multihashes TBD Migration of Blockstore to use multihash instead of CID as key #2415