Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CLI for migration #1035

Merged
merged 15 commits into from
Oct 22, 2024
Merged

Add CLI for migration #1035

merged 15 commits into from
Oct 22, 2024

Conversation

JOOHOJANG
Copy link
Contributor

@JOOHOJANG JOOHOJANG commented Oct 16, 2024

What this PR does / why we need it:
Define semi-auto migration architecture and add migration scripts for add version vector.

Process

1. make new version directory in migrations
2. add migration script files.
3. add version into migrationMap in /cmd/migration.go

Usage example

$ ./yorkie migration --from v0.5.1 --to v0.6.0 --mongo-connection-uri mongodb://localhost:27017 --batch-size 10000 --database-name yorkie-meta

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:


Additional documentation:


Checklist:

  • Added relevant tests or not required
  • Didn't break anything

@JOOHOJANG JOOHOJANG requested a review from hackerwins October 16, 2024 00:22
@JOOHOJANG JOOHOJANG self-assigned this Oct 16, 2024
@JOOHOJANG JOOHOJANG removed the request for review from hackerwins October 16, 2024 00:22
@JOOHOJANG JOOHOJANG marked this pull request as ready for review October 16, 2024 03:12
@JOOHOJANG JOOHOJANG requested a review from hackerwins October 18, 2024 07:43
@hackerwins hackerwins changed the base branch from hybrid-clock to main October 18, 2024 08:13
@hackerwins hackerwins changed the base branch from main to hybrid-clock October 18, 2024 08:14
Copy link
Member

@hackerwins hackerwins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your contribution. Overall looks good. I have a few questions.

A. migrations directory

Currently, Yorkie is built and distributed as a single binary(yorkie). In this change, it seems that Yorkie CLI is designed to execute by referencing the migrations directory.

While migrations directory is included in the binary during the build, it is unclear why the command(yorkie migration) still needs to reference the directory.

It would be better to retrieve and execute the target scripts based on the migrationMap rather than external directory. I'm curious about your opinion.

B. Batch Processing for Find

Please, consider implementing batch processing for Find. The current approach of loading the entire collection into memory could lead to excessive memory usage.

collection := db.Database(databaseName).Collection("changes")

cursor, err := collection.Find(ctx, bson.M{})
if err != nil {
  return err
}

@JOOHOJANG
Copy link
Contributor Author

Thanks for your contribution. Overall looks good. I have a few questions.

A. migrations directory

Currently, Yorkie is built and distributed as a single binary(yorkie). In this change, it seems that Yorkie CLI is designed to execute by referencing the migrations directory.

While migrations directory is included in the binary during the build, it is unclear why the command(yorkie migration) still needs to reference the directory.

It would be better to retrieve and execute the target scripts based on the migrationMap rather than external directory. I'm curious about your opinion.

B. Batch Processing for Find

Please, consider implementing batch processing for Find. The current approach of loading the entire collection into memory could lead to excessive memory usage.

collection := db.Database(databaseName).Collection("changes")

cursor, err := collection.Find(ctx, bson.M{})
if err != nil {
  return err
}

A. Agree, you're right. Watching migration map is enough. I'll fix it.
B. Cursor has default batch size (101 documents) so �we should not increase it unless we need a larger batch size. https://www.mongodb.com/docs/manual/tutorial/iterate-a-cursor/

@JOOHOJANG JOOHOJANG requested a review from hackerwins October 21, 2024 12:33
Copy link
Member

@hackerwins hackerwins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your contribution.

@hackerwins hackerwins merged commit 88735cc into hybrid-clock Oct 22, 2024
1 check passed
@hackerwins hackerwins deleted the add-cli-migration branch October 22, 2024 01:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants