1.5 Akka.Cluster.Sharding migration documentation (#6124)

* added detailed Akka.NET v1.5 sharding migration guide

* close #6123

added separate documentation for #6123

docs/articles/clustering/cluster-sharding.md

@@ -6,14 +6,17 @@ title: Akka.Cluster.Sharding module
 
 Cluster sharding is useful in cases when you want to contact with cluster actors using their logical id's, but don't want to care about their physical location inside the cluster or manage their creation. Moreover it's able to re-balance them, as nodes join/leave the cluster. It's often used to represent i.e. Aggregate Roots in Domain Driven Design terminology.
 
+> [!IMPORTANT]
+> Interested in upgrading an Akka.NET v1.4 Cluster.Sharding application to v1.5? [Please read our Akka.Cluster.Sharding v1.5 migration guide](xref:akkadotnet-v15-upgrade-advisories#akkaclustersharding-state-storage).
+
 Cluster sharding can operate in 2 modes, configured via `akka.cluster.sharding.state-store-mode` HOCON configuration:
 
 1. `persistence` (**default**) depends on Akka.Persistence module. In order to use it, you'll need to specify an event journal accessible by all of the participating nodes. An information about the particular shard placement is stored in a persistent cluster singleton actor known as *coordinator*. In order to guarantee consistent state between different incarnations, coordinator stores its own state using Akka.Persistence event journals.
 2. `ddata` depends on Akka.DistributedData module. It uses Conflict-free Replicated Data Types (CRDT) to ensure eventually consistent shard placement and global availability via node-to-node replication and automatic conflict resolution. In this mode event journals don't have to be configured.
 
 Cluster sharding may be active only on nodes in `Up` status - so the ones fully recognized and acknowledged by every other node in a cluster.
 
-## QuickStart
+## Quick Start
 
 Actors managed by cluster sharding are called **entities** and can be automatically distributed across multiple nodes inside the cluster. One entity instance may live only at one node at the time, and can be communicated with via `ShardRegion` without need to know, what it's exact node location is.
 
@@ -135,6 +138,7 @@ akka.cluster.sharding.state-store-mode = persistence
 ```
 
 This mode uses [persistence](../persistence/event-sourcing.md) to store the active shards and active entities for each shard.
+
 By default, cluster sharding will use the journal and snapshot store plugin defined in `akka.persistence.journal.plugin` and
 `akka.persistence.snapshot-store.plugin` respectively; to change this behavior, you can use these configuration:
 
@@ -143,6 +147,65 @@ akka.cluster.sharding.journal-plugin-id = <plugin>
 akka.cluster.sharding.snapshot-plugin-id = <plugin>
 ```
 
+> [!IMPORTANT]
+> It's considered a good practice to have Akka.Cluster.Sharding store its state in a separate journal and snapshot store - that way, in the event that you need to purge all sharding data, this can be easily isolated in its own table.
+
+You can have Akka.Cluster.Sharding use its own separate journal and snapshot store via the following HOCON, for instance:
+
+```hocon
+akka.persistence {
+    # default plugins
+    journal {
+        plugin = "akka.persistence.journal.mongodb"
+        mongodb {
+            # qualified type name of the MongoDb persistence journal actor
+            class = "Akka.Persistence.MongoDb.Journal.MongoDbJournal, Akka.Persistence.MongoDb"
+
+            # connection string used for database access
+            connection-string = ""
+            collection = "EventJournal"
+            metadata-collection = "Metadata"
+        }
+
+        sharding {
+            # qualified type name of the MongoDb persistence journal actor
+            class = "Akka.Persistence.MongoDb.Journal.MongoDbJournal, Akka.Persistence.MongoDb"
+
+            # connection string used for database access
+            connection-string = ""
+
+            # separate collections / tables for Akka.Cluster.Sharding
+            collection = "EventJournalSharding"
+            metadata-collection = "MetadataSharding"
+        }
+    }
+
+    snapshot-store {
+        plugin = "akka.persistence.snapshot-store.mongodb"
+        mongodb {
+            class = "Akka.Persistence.MongoDb.Snapshot.MongoDbSnapshotStore, Akka.Persistence.MongoDb"
+
+            # connection string used for database access
+            connection-string = ""
+
+            collection = "SnapshotStore"
+        }
+
+        sharding {
+            class = "Akka.Persistence.MongoDb.Snapshot.MongoDbSnapshotStore, Akka.Persistence.MongoDb"
+
+            # connection string used for database access
+            connection-string = ""
+
+            collection = "SnapshotStoreSharding"
+        }
+    }
+}
+
+akka.cluster.sharding.journal-plugin-id = akka.persistence.journal.sharding
+akka.cluster.sharding.snapshot-plugin-id = akka.persistence.snapshot-store.sharding
+```
+
 #### Remember Entities Distributed Data Mode
 
 You can enable DData mode by setting these configuration:
@@ -151,8 +214,7 @@ You can enable DData mode by setting these configuration:
 akka.cluster.sharding.state-store-mode = ddata
 ```
 
-To support restarting entities after a full cluster restart (non-rolling) the remember entities store
-is persisted to disk by distributed data. This can be disabled if not needed:
+To support restarting entities after a full cluster restart (non-rolling) the remember entities store is persisted to disk by distributed data. This can be disabled if not needed:
 
 ```hocon
 akka.cluster.sharding.distributed-data.durable.keys = []

docs/articles/persistence/event-sourcing.md

@@ -1,6 +1,6 @@
 ---
 uid: persistent-actors
-title: Event sourcing
+title: Event Sourcing with Akka.Persistence Actors
 ---
 # Event Sourcing
 

docs/cSpell.json

@@ -28,6 +28,8 @@
         "deserialization",
         "downstream",
         "downstreams",
+        "ddata",
+        "eventsourced",
         "Gitter",
         "Hasher",
         "Hipsterize",

docs/community/whats-new/akkadotnet-v1.5-upgrade-advisories.md

@@ -0,0 +1,162 @@
+---
+uid: akkadotnet-v15-upgrade-advisories
+title: Akka.NET v1.5 Upgrade Advisories
+---
+
+# Akka.NET v1.5 Upgrade Advisories
+
+This document contains specific upgrade suggestions, warnings, and notices that you will want to pay attention to when upgrading between versions within the Akka.NET v1.5 roadmap.
+
+## Upgrading From Akka.NET v1.4 to v1.5
+
+### Akka.Cluster.Sharding State Storage
+
+One of the most significant upgrades we've made in Akka.NET v1.5 is a complete rewrite of Akka.Cluster.Sharding's state storage system.
+
+> [!NOTE]
+> You can watch [our discussion of this Akka.Cluster.Sharding upgrade during our September, 2022 Akka.NET Community Standup for more details](https://www.youtube.com/watch?v=rTBgxeHf91M&t=359s).
+
+In Akka.NET v1.5 we've split Akka.Cluster.Sharding's `state-store-mode` into two parts:
+
+* CoordinatorStore (`akka.cluster.sharding.state-store-mode`) and
+* ShardStore (`akka.cluster.sharding.remember-entities-store`.)
+
+Which can use different persistence modes configured via `akka.cluster.sharding.state-store-mode` & `akka.cluster.sharding.remember-entities-store`.
+
+> [!IMPORTANT]
+> The goal behind this split was to remove the `ShardCoordinator` as a single point of bottleneck during `remember-entities=on` recovery - in Akka.NET v1.4 all remember-entity state is concentrated in the journal of the `PersistentShardCoordinator` or the CRDT used by the `DDataCoordinator`. In v1.5 the responsibility for remembering entities has been pushed to the `Shard` actors themselves, which allows for remembered-entities to be recovered in parallel for all shards.
+
+Possible combinations:
+
+state-store-mode | remember-entities-store | CoordinatorStore mode | ShardStore mode
+------------------ | ------------------------- | ------------------------ | ------------------
+persistence (default) | - (ignored) | persistence | persistence
+ddata | ddata | ddata | ddata
+ddata | eventsourced (new) | ddata | persistence
+
+There should be no breaking changes from user perspective. Only some internal messages/objects were moved. There should be no change in the `PersistentId` behavior and default persistent configuration (`akka.cluster.sharding.state-store-mode`)
+
+This change is designed to speed up the performance of Akka.Cluster.Sharding coordinator recovery by moving `remember-entities` recovery into separate actors - this also solves major performance problems with the `ddata` recovery mode overall.
+
+The recommended settings for maximum ease-of-use for Akka.Cluster.Sharding in new applications going forward will be:
+
+```hocon
+akka.cluster.sharding{
+  state-store-mode = ddata
+  remember-entities-store = eventsourced
+}
+```
+
+However, for the sake of backwards compatibility the Akka.Cluster.Sharding defaults have been left as-is:
+
+```hocon
+akka.cluster.sharding{
+  state-store-mode = persistence
+  # remember-entities-store (not set - also uses legacy Akka.Persistence)
+}
+```
+
+#### Migrating to New Sharding Storage From Akka.Persistence
+
+> [!NOTE]
+> This section applies only to users who were using `akka.cluster.sharding.state-store-mode = persistence`. If you were using `akka.cluster.sharding.state-store-mode`
+
+Switching over to using `remember-entities-store = eventsourced` will cause an initial migration of data from the `ShardCoordinator`'s journal into separate event journals going forward.
+
+Upgrading to Akka.NET v1.5 will **cause an irreversible migration of Akka.Cluster.Sharding data** for users who were previously running `akka.cluster.state-store-mode=persistence`, so follow the steps below carefully:
+
+> [!IMPORTANT]
+> This migration is intended to be performed via upgrading Akka.NET to v1.5 and applying HOCON configuration changes - it requires no downtime.
+
+##### Step 1 - Upgrade to Akka.NET v1.5 With Updated Persistence HOCON
+
+Update your Akka.Cluster.Sharding HOCON to look like the following (adjust as necessary for your custom settings):
+
+```hocon
+akka.cluster.sharding {
+    remember-entities = on
+    remember-entities-store = "eventsourced"
+    state-store-mode = "persistence"
+
+    # fail if upgrade doesn't succeed
+    fail-on-invalid-entity-state-transition = on
+}
+
+ akka.persistence.journal.{your-journal-implementation} {
+    event-adapters {
+        coordinator-migration = ""Akka.Cluster.Sharding.OldCoordinatorStateMigrationEventAdapter, Akka.Cluster.Sharding""
+    }
+
+    event-adapter-bindings {
+        ""Akka.Cluster.Sharding.ShardCoordinator+IDomainEvent, Akka.Cluster.Sharding"" = coordinator-migration
+    }
+}
+```
+
+> [!NOTE]
+> If you don't run Akka.Cluster.Sharding with `remember-entities=on` normally then _there is no need to turn it on here_.
+
+With these HOCON settings in-place the following will happen:
+
+1. The old `PersitentShardCoordinator` state will be broken up - `remember-entities=on` data will be distributed to each of the `PersistentShard` actors, who will now use the new `remember-entities-store = "eventsourced"` setting going forward;
+2. Old `Akka.Cluster.Sharding.ShardCoordinator+IDomainEvent` will be upgraded to a new storage format via the `coordinator-migration` Akka.Persistence event adapter; and
+3. The `PersistentShardCoordinator` will migrate its journal to the new format as well.
+
+##### Step 2 - Migrating Away From Persistence to DData
+
+Once your cluster has successfully booted up with these settings, you can now optionally move to using `DData` as your `akka.cluster.sharding.state-store-mode` by deploying a second time with the following HOCON:
+
+```hocon
+akka.cluster.sharding {
+    remember-entities = on
+    remember-entities-store = "eventsourced"
+    state-store-mode = "ddata"
+
+    # fail if upgrade doesn't succeed
+    fail-on-invalid-entity-state-transition = on
+}
+
+ akka.persistence.journal.{your-journal-implementation} {
+    event-adapters {
+        coordinator-migration = ""Akka.Cluster.Sharding.OldCoordinatorStateMigrationEventAdapter, Akka.Cluster.Sharding""
+    }
+
+    event-adapter-bindings {
+        ""Akka.Cluster.Sharding.ShardCoordinator+IDomainEvent, Akka.Cluster.Sharding"" = coordinator-migration
+    }
+}
+```
+
+Now you'll be running Akka.Cluster.Sharding with the recommended settings.
+
+#### Migrating to New Sharding Storage From Akka.DistributedData
+
+The migration process onto Akka.NET v1.5's new Cluster.Sharding storage system is less involved for users who were already using `akka.cluster.sharding.state-store-mode=ddata`.
+
+All these users need to do this:
+
+1. Setup an `akka.persistence.journal` and `akka.persistence.snapshot-store` to use with `akka.cluster.sharding.remember-entities-store = eventsourced`;
+2. Deploy using the following HOCON:
+
+```hocon
+akka.cluster.sharding {
+    remember-entities = on
+    remember-entities-store = "eventsourced"
+    state-store-mode = "ddata"
+
+    # fail if upgrade doesn't succeed
+    fail-on-invalid-entity-state-transition = on
+}
+
+ akka.persistence.journal.{your-journal-implementation} {
+    event-adapters {
+        coordinator-migration = ""Akka.Cluster.Sharding.OldCoordinatorStateMigrationEventAdapter, Akka.Cluster.Sharding""
+    }
+
+    event-adapter-bindings {
+        ""Akka.Cluster.Sharding.ShardCoordinator+IDomainEvent, Akka.Cluster.Sharding"" = coordinator-migration
+    }
+}
+```
+
+If you run into any trouble upgrading, [please file an issue with Akka.NET](https://github.com/akkadotnet/akka.net/issues/new/choose).

docs/community/whats-new/akkadotnet-v1.5.md

@@ -47,6 +47,56 @@ Contributing to the v1.5 effort is somewhat different than our [normal maintenan
 
 ## Designs & Major Changes
 
+### Akka.Cluster.Sharding State Storage
+
+One of the most significant upgrades we've made in Akka.NET v1.5 is a complete rewrite of Akka.Cluster.Sharding's state storage system.
+
+> [!NOTE]
+> You can watch [our discussion of this Akka.Cluster.Sharding upgrade during our September, 2022 Akka.NET Community Standup for more details](https://www.youtube.com/watch?v=rTBgxeHf91M&t=359s).
+
+In Akka.NET v1.5 we've split Akka.Cluster.Sharding's `state-store-mode` into two parts:
+
+* CoordinatorStore (`akka.cluster.sharding.state-store-mode`) and
+* ShardStore (`akka.cluster.sharding.remember-entities-store`.)
+
+Which can use different persistence modes configured via `akka.cluster.sharding.state-store-mode` & `akka.cluster.sharding.remember-entities-store`.
+
+> [!IMPORTANT]
+> The goal behind this split was to remove the `ShardCoordinator` as a single point of bottleneck during `remember-entities=on` recovery - in Akka.NET v1.4 all remember-entity state is concentrated in the journal of the `PersistentShardCoordinator` or the CRDT used by the `DDataCoordinator`. In v1.5 the responsibility for remembering entities has been pushed to the `Shard` actors themselves, which allows for remembered-entities to be recovered in parallel for all shards.
+
+Possible combinations:
+
+state-store-mode | remember-entities-store | CoordinatorStore mode | ShardStore mode
+------------------ | ------------------------- | ------------------------ | ------------------
+persistence (default) | - (ignored) | persistence | persistence
+ddata | ddata | ddata | ddata
+ddata | eventsourced (new) | ddata | persistence
+
+There should be no breaking changes from user perspective. Only some internal messages/objects were moved. There should be no change in the `PersistentId` behavior and default persistent configuration (`akka.cluster.sharding.state-store-mode`)
+
+This change is designed to speed up the performance of Akka.Cluster.Sharding coordinator recovery by moving `remember-entities` recovery into separate actors - this also solves major performance problems with the `ddata` recovery mode overall.
+
+The recommended settings for maximum ease-of-use for Akka.Cluster.Sharding in new applications going forward will be:
+
+```hocon
+akka.cluster.sharding{
+  state-store-mode = ddata
+  remember-entities-store = eventsourced
+}
+```
+
+However, for the sake of backwards compatibility the Akka.Cluster.Sharding defaults have been left as-is:
+
+```hocon
+akka.cluster.sharding{
+  state-store-mode = persistence
+  # remember-entities-store (not set - also uses legacy Akka.Persistence)
+}
+```
+
+> [!IMPORTANT]
+> Read "[Upgrading to Akka.NET V1.5 - Cluster.Sharding](xref:akkadotnet-v15-upgrade-advisories)" for details on how to upgrade from Akka.NET v1.5.
+
 ### Akka.Hosting
 
 We want to make Akka.NET something that can be instantiated more typically per the patterns often used with the Microsoft.Extensions.Hosting APIs that are common throughout .NET.

docs/community/whats-new/toc.yml

@@ -1,5 +1,7 @@
 - name: New in Akka.NET v1.5
   href: akkadotnet-v1.5.md
+- name: Upgrade Advisories (v1.5)
+  href: akkadotnet-v1.5-upgrade-advisories.md
 - name: New in Akka.NET v1.4
   href: akkadotnet-v1.4.md
 - name: Upgrade Advisories (v1.4)

src/contrib/cluster/Akka.Cluster.Sharding.Tests/PersistentShardingMigrationSpec.cs

@@ -114,6 +114,17 @@ class = ""Akka.Cluster.Sharding.Tests.MemorySnapshotStoreShared, Akka.Cluster.Sh
                     retry-interval = 500ms
                 }
                 akka.cluster.sharding.fail-on-invalid-entity-state-transition = on
+
+                 akka.persistence.journal.memory-journal-shared {
+                    event-adapters {
+                        coordinator-migration = ""Akka.Cluster.Sharding.OldCoordinatorStateMigrationEventAdapter, Akka.Cluster.Sharding""
+                    }
+
+                    event-adapter-bindings {
+                        ""Akka.Cluster.Sharding.ShardCoordinator+IDomainEvent, Akka.Cluster.Sharding"" = coordinator-migration
+                    }
+                }
+
                 akka.cluster.sharding.verbose-debug-logging = on")
                     .WithFallback(ClusterSingletonManager.DefaultConfig())
                     .WithFallback(ClusterSharding.DefaultConfig());