Migration tools to migrate serialized data from 3.x to 7.0 #8214
Conversation
benjaminpetit
force-pushed
the
proto/state-migration
branch
from
ba74352 to
9d939d3
Compare
benjaminpetit
force-pushed
the
proto/state-migration
branch
from
500bf71 to
e7c4ebf
Compare
|
What needs to be done before this is ready for review? |
| /// Get all entries in storage | ||
| /// </summary> | ||
| /// <returns>The entries in storage</returns> | ||
| IAsyncEnumerable<StorageEntry> GetAll(CancellationToken cancellationToken); |
There was a problem hiding this comment.
Should this cancellation token be marked with the [EnumeratorCancellation] attribute?
There was a problem hiding this comment.
I wonder if there should be an optional "fromKey" parameter so that enumeration can skip already-processed keys. Maybe it's not necessary at this stage.
There was a problem hiding this comment.
I think ReadAllAsync would align better with other method names
There was a problem hiding this comment.
I think we dont need it on the interface type, since it requires the specific implementation of the method:
So we should use it on every single method implementation, but not on the interface level. Also @benjaminpetit added it in the Orleans.Storage.AzureBlobGrainStorage in example
| /// <param name="input">The data to deserialize.</param> | ||
| /// <typeparam name="T">The output type.</typeparam> | ||
| /// <returns>The deserialized object.</returns> | ||
| T Deserialize<T>(BinaryData input); |
There was a problem hiding this comment.
I wonder if we need to solve the issue of string vs binary data here. Eg, for JSON-based storage systems like Postgres (json columns), Mongo DB, & CosmosDB
| /// <inheritdoc/> | ||
| public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer) | ||
| { | ||
| throw new NotImplementedException(); |
There was a problem hiding this comment.
We probably want to delete these commented-out bits before merge.
| { | ||
| private readonly string _filename; | ||
|
|
||
| public CsvDataReader(string filename) |
|
|
||
| [SkippableTheory] | ||
| [CsvDataReader("grain-references.csv")] | ||
| public void GrainRefenceSerializerTest(Type grainInterfaceType, string key, string keyExt, string expected) |
There was a problem hiding this comment.
How do we re-generate this csv, should that be documented?
|
How far are we from have this? We are already talking about Orleans 8 but we can't move to 7 because of the storage/reminders migration. |
| public async IAsyncEnumerable<StorageEntry> GetAll([EnumeratorCancellation] CancellationToken cancellationToken) | ||
| { | ||
| var regex = new Regex("(?<name>[^-]+)-(?<reference>[^-]+).json"); | ||
| await foreach (var item in this.container.GetBlobsAsync(cancellationToken: cancellationToken)) |
There was a problem hiding this comment.
We probably need a way to store progress periodically so that we make forward progress after a failure. Eg, store the last migrated key periodically and scan blob names starting from that id on recovery.
There was a problem hiding this comment.
It may also be advantageous to support a graceful "pausing" of the migration. Some environments may only want to run this during periods of lower systems load.
|
This pull request has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 7 days. It will be closed if no further activity occurs within 7 days of this comment. |
|
bad bot! |
|
This pull request has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 7 days. It will be closed if no further activity occurs within 7 days of this comment. |
ghost
closed this
|
What happened? Bot went rogue? |
|
Bad bot! |
|
we need to support translation of the Reminders as well (done by Benjamin in src/Orleans.Persistence.Migration/GrainReferenceExtractor.cs, I just need to connect it in the IReminderTable) |
|
supported reminders and finalized the grains migrations here |
The goal of this PR is to introduce a set of tools to enable data migration from 3.x to 7.0.
For the moment:
Microsoft Reviewers: Open in CodeFlow