Configuration Options

This page outlines the configuration options for three key migrations:

1. Metadata Migration
2. Backfill Migration with Reindex-from-Snapshot (RFS)
3. Live Capture Migration with Capture and Replay (C&R)

Each of these migrations may depend on either a snapshot or a capture proxy. The CDK context blocks below are shown as separate context blocks for each migration type for simplicity. If performing multiple migration types, combine these options, as the actual execution of each migration is controlled from the Migration Console.

Tip

For a complete list of configuration options, please refer to the opensearch-migrations options.md but please open an issue for consultation if changing an option that is not listed on this page.

Options for the source cluster endpoint, target cluster endpoint, and existing VPC should be configured for the Migration tools to function effectively.

Tip

The CDK context blocks below are shown as separate context blocks for each migration type for simplicity. If performing multiple migration types, combine these options, as the actual execution of each migration is controlled from the Migration Console.

Metadata Migration Options

Sample Metadata Migration CDK Options

{
  "metadata-migration": {
    "stage": "dev",
    "vpcId": <VPC_ID>,
    "sourceClusterEndpoint": <SOURCE_CLUSTER_ENDPOINT>,
    "targetClusterEndpoint": <TARGET_CLUSTER_ENDPOINT>,
    "reindexFromSnapshotServiceEnabled": true,
    "artifactBucketRemovalPolicy": "DESTROY"
  }
}

There are currently no CDK options specific to Metadata migrations, which are performed from the Migration Console. This migration requires an existing snapshot, which can be created from the Migration Console.

Shared configuration options table

Name	Example	Description
sourceClusterEndpoint	"https://source-cluster.elb.us-east-1.endpoint.com"	The endpoint for the source cluster.
targetClusterEndpoint	"https://vpc-demo-opensearch-cluster-cv6hggdb66ybpk4kxssqt6zdhu.us-west-2.es.amazonaws.com:443"	The endpoint for the target cluster. Required if using an existing target cluster for the migration instead of creating a new one.
vpcId	"vpc-123456789abcdefgh"	The ID of the existing VPC where the migration resources will be placed. The VPC must have at least two private subnets that span two availability zones.

Backfill Migration with Reindex-from-Snapshot (RFS) Options

Sample Backfill Migration CDK Options

{
  "backfill-migration": {
    "stage": "dev",
    "vpcId": <VPC_ID>,
    "sourceClusterEndpoint": <SOURCE_CLUSTER_ENDPOINT>,
    "targetClusterEndpoint": <TARGET_CLUSTER_ENDPOINT>,
    "reindexFromSnapshotServiceEnabled": true,
    "reindexFromSnapshotExtraArgs": "",
    "artifactBucketRemovalPolicy": "DESTROY"
  }
}

Performing a Reindex-from-Snapshot backfill migration requires an existing snapshot. The CDK options specific to backfill migrations are listed below. To view all available arguments for reindexFromSnapshotExtraArgs, see here. At a minimum, no extra arguments may be needed.

Backfill specific configuration options table

Name	Example	Description
reindexFromSnapshotServiceEnabled	true	Enables deploying and configuring the RFS ECS service.
reindexFromSnapshotExtraArgs	"--target-aws-region us-east-1 --target-aws-service-signing-name es"	Extra arguments for the Document Migration command, with space separation. See the RFS Extra Arguments for more details. You can pass --no-insecure to remove the --insecure flag.

Live Capture Migration with Capture and Replay (C&R) Options

Sample Live Capture Migration CDK Options

{
  "live-capture-migration": {
    "stage": "dev",
    "vpcId": <VPC_ID>,
    "sourceClusterEndpoint": <SOURCE_CLUSTER_ENDPOINT>,
    "targetClusterEndpoint": <TARGET_CLUSTER_ENDPOINT>,
    "captureProxyServiceEnabled": true,
    "captureProxyExtraArgs": "",
    "trafficReplayerServiceEnabled": true,
    "trafficReplayerExtraArgs": "",
    "artifactBucketRemovalPolicy": "DESTROY"
  }
}

Performing a live capture migration requires that a Capture Proxy be configured to capture incoming traffic and send it to the target cluster via the Traffic Replayer service. For arguments available in captureProxyExtraArgs, refer to the @Parameter fields here. For trafficReplayerExtraArgs, refer to the @Parameter fields here. At a minimum, no extra arguments may be needed.

Capture and Replay specific configuration options table

Name	Example	Description
captureProxyServiceEnabled	true	Enables the Capture Proxy service deployment via a new CloudFormation stack.
captureProxyExtraArgs	"--suppressCaptureForHeaderMatch user-agent .*elastic-java/7.17.0.*"	Extra arguments for the Capture Proxy command, including options specified by the Capture Proxy.
trafficReplayerServiceEnabled	true	Enables the Traffic Replayer service deployment via a new CloudFormation stack.
trafficReplayerExtraArgs	"--sigv4-auth-header-service-region es,us-east-1 --speedup-factor 5"	Extra arguments for the Traffic Replayer command, including options for auth headers and other parameters specified by the Traffic Replayer.

Troubleshooting

Network Configuration

The migration tooling expects the source cluster, target cluster, and migration resources to exist in the same VPC. If this is not the case, manual networking setup outside of this documentation is likely required.

TODOs

TODO: Remove requirement that Metadata migration requires the RFS service be enabled

TODO: Make Replayer args clearer in a README file

TODO: Present a clear path for what options need to be provided for different auth patterns (no auth, basic auth, sigv4) that are used for source/target cluster