Configuration Options

This page lays out configuration options for three key migrations:

1. Metadata migration
2. Document migration
3. Live Traffic replication

Notice that each of these migrations may be dependent upon either a snapshot or capture proxy. Also note that other than the snapshot/proxy prerequisite, each of these three migrations are independent of each other. For a complete list of options, please refer to opensearch-migrations options.md

Regardless of the type of migration, options for the source and target that are shown below should be configured for the Migration tools to perform the necessary migration

Name	Example	Description
sourceClusterEndpoint	"https://source-cluster.elb.us-east-1.endpoint.com"	The endpoint for the source cluster from which RFS will take a snapshot
targetClusterEndpoint	"https://vpc-demo-opensearch-cluster-cv6hggdb66ybpk4kxssqt6zdhu.us-west-2.es.amazonaws.com:443"	The target cluster endpoint which will be used to replay captured traffic against. Required if using an existing target cluster for the migration, instead of creating one. This endpoint must be in the same VPC as the Migration services to allow for proper linking

Metadata Migration Options

Sample Metadata Migration CDK Options

{
  "metadata-migration": {
    "stage": "dev",
    "sourceClusterEndpoint": <SOURCE_CLUSTER_ENDPOINT>,
    "targetClusterEndpoint": <TARGET_CLUSTER_ENDPOINT>,
    "reindexFromSnapshotServiceEnabled": true,
    "artifactBucketRemovalPolicy": "DESTROY"
  }
}

There are no CDK options, currently, for configuring Metadata migrations specifically, which are performed from the Migration Console. This migration requires an existing snapshot, which can be created when accessing the Migration Console.

Document Migration Options

Sample Document Migration CDK Options

{
  "document-migration": {
    "stage": "dev",
    "sourceClusterEndpoint": <SOURCE_CLUSTER_ENDPOINT>,
    "targetClusterEndpoint": <TARGET_CLUSTER_ENDPOINT>,
    "reindexFromSnapshotServiceEnabled": true,
    "reindexFromSnapshotExtraArgs": "",
    "artifactBucketRemovalPolicy": "DESTROY"
  }
}

The specific CDK options for document migrations are below. To find all available arguments that can be provided to reindexFromSnapshotExtraArgs see here. At a minimum no extra arguments may be needed.

Name	Example	Description
reindexFromSnapshotServiceEnabled	true	Create resources for deploying and configuring the RFS ECS service
reindexFromSnapshotExtraArgs	"--target-aws-region us-east-1 --target-aws-service-signing-name es"	Extra arguments to provide to the Document Migration command with space separation. See RFS Extra Arguments below for available options. Boolean negation is used, so --no-insecure can be passed to remove the --insecure flag if present.

Live Traffic Replication Options

Sample Live Traffic Migration CDK Options

{
  "live-traffic-migration": {
    "stage": "dev",
    "sourceClusterEndpoint": <SOURCE_CLUSTER_ENDPOINT>,
    "targetClusterEndpoint": <TARGET_CLUSTER_ENDPOINT>,
    "trafficReplayerServiceEnabled": true,
    "trafficReplayerExtraArgs": "",
    "artifactBucketRemovalPolicy": "DESTROY"
  }
}

To find all available arguments that can be provided to trafficReplayerExtraArgs see the @Parameter fields here. At a minimum no extra arguments may be needed.

Name	Example	Description
trafficReplayerServiceEnabled	true	Enable deploying the given service, via a new CloudFormation stack
trafficReplayerExtraArgs	"--sigv4-auth-header-service-region es,us-east-1 --speedup-factor 5"	Extra arguments to provide to the Replayer command. This includes auth header options and other parameters supported by the Traffic Replayer.

TODOs

TODO: Remove requirement that Metadata migration requires the RFS service be enabled

TODO: Make Replayer args clearer in a README file

TODO: Present a clear path for what options need to be provided for different auth patterns (no auth, basic auth, sigv4) that are used for source/target cluster

Owner: @lewijacn