-
Notifications
You must be signed in to change notification settings - Fork 130
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Make the search port configurable so that we can avoid conflicts be…
…tween the ElasticSearch and OpenSearch integration tests - Add documentation on how to migrate from ElasticSearch to OpenSearch (not tested yet)
- Loading branch information
1 parent
43016ca
commit e2125d6
Showing
6 changed files
with
248 additions
and
10 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,6 @@ | ||
| /NOTICE-generated | ||
| .settings | ||
| .vscode | ||
| target | ||
| .classpath | ||
| .project | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
212 changes: 212 additions & 0 deletions
212
manual/src/main/asciidoc/migrations/migrate-elasticsearch-to-opensearch.adoc
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,212 @@ | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); | ||
| // you may not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, software | ||
| // distributed under the License is distributed on an "AS IS" BASIS, | ||
| // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| // See the License for the specific language governing permissions and | ||
| // limitations under the License. | ||
| // | ||
|
|
||
| = Migrating from Elasticsearch to OpenSearch | ||
| :toc: macro | ||
| :toclevels: 4 | ||
| :toc-title: Table of contents | ||
| :numbered: | ||
|
|
||
| toc::[] | ||
|
|
||
| == Overview | ||
|
|
||
| This guide describes how to migrate your Apache Unomi data from Elasticsearch to OpenSearch. The migration process involves using the OpenSearch Replication Tool, which is designed to handle large-scale migrations efficiently while maintaining data consistency. | ||
|
|
||
| == Prerequisites | ||
|
|
||
| Before starting the migration, ensure you have: | ||
|
|
||
| * Running Elasticsearch cluster with your Unomi data | ||
| * Target OpenSearch cluster set up and running | ||
| * Sufficient disk space on the target cluster | ||
| * Java 11 or later installed | ||
| * Network connectivity between source and target clusters | ||
|
|
||
| == Migration Options | ||
|
|
||
| === Option 1: OpenSearch Replication Tool (Recommended) | ||
|
|
||
| The OpenSearch Replication Tool is the recommended approach for production environments, especially for large datasets. | ||
|
|
||
| ==== Installation | ||
|
|
||
| [source,bash] | ||
| ---- | ||
| git clone https://github.com/opensearch-project/opensearch-migrations.git | ||
| cd opensearch-migrations/replication-tool | ||
| ./gradlew build | ||
| ---- | ||
|
|
||
| ==== Configuration | ||
|
|
||
| Create a configuration file `config.yml`: | ||
|
|
||
| [source,yaml] | ||
| ---- | ||
| source: | ||
| hosts: ["source-elasticsearch-host:9200"] | ||
| user: "elastic_user" # if authentication is enabled | ||
| password: "elastic_pass" # if authentication is enabled | ||
| destination: | ||
| hosts: ["target-opensearch-host:9200"] | ||
| user: "opensearch_user" # if authentication is enabled | ||
| password: "opensearch_pass" # if authentication is enabled | ||
| indices: | ||
| - name: "context-*" # Unomi context indices | ||
| - name: "segment-*" # Unomi segment indices | ||
| - name: "profile-*" # Unomi profile indices | ||
| - name: "session-*" # Unomi session indices | ||
| ---- | ||
|
|
||
| ==== Running the Migration | ||
|
|
||
| [source,bash] | ||
| ---- | ||
| ./bin/replication-tool --config config.yml | ||
| ---- | ||
|
|
||
| The tool provides progress updates and ensures data consistency during the migration. | ||
|
|
||
| === Option 2: Logstash Pipeline | ||
|
|
||
| For smaller deployments or when more control over the migration process is needed, you can use Logstash. | ||
|
|
||
| ==== Logstash Configuration | ||
|
|
||
| Create a file named `logstash-migration.conf`: | ||
|
|
||
| [source,ruby] | ||
| ---- | ||
| input { | ||
| elasticsearch { | ||
| hosts => ["source-elasticsearch-host:9200"] | ||
| index => "context-*" # Repeat for other indices | ||
| size => 5000 | ||
| scroll => "5m" | ||
| docinfo => true | ||
| user => "elastic_user" # if authentication is enabled | ||
| password => "elastic_pass" # if authentication is enabled | ||
| } | ||
| } | ||
| output { | ||
| opensearch { | ||
| hosts => ["target-opensearch-host:9200"] | ||
| index => "%{[@metadata][_index]}" | ||
| document_id => "%{[@metadata][_id]}" | ||
| user => "opensearch_user" # if authentication is enabled | ||
| password => "opensearch_pass" # if authentication is enabled | ||
| } | ||
| } | ||
| ---- | ||
|
|
||
| ==== Running Logstash Migration | ||
|
|
||
| [source,bash] | ||
| ---- | ||
| logstash -f logstash-migration.conf | ||
| ---- | ||
|
|
||
| == Post-Migration Steps | ||
|
|
||
| 1. Verify Data Integrity | ||
| + | ||
| [source,bash] | ||
| ---- | ||
| # Check document counts | ||
| curl -X GET "source-elasticsearch-host:9200/_cat/indices/context-*?v" | ||
| curl -X GET "target-opensearch-host:9200/_cat/indices/context-*?v" | ||
| ---- | ||
|
|
||
| 2. Update Unomi Configuration | ||
| + | ||
| Edit `etc/custom.system.properties`: | ||
| + | ||
| [source,properties] | ||
| ---- | ||
| # Comment out or remove Elasticsearch properties | ||
| #org.apache.unomi.elasticsearch.addresses=localhost:9200 | ||
| #org.apache.unomi.elasticsearch.cluster.name=contextElasticSearch | ||
| # Add OpenSearch properties | ||
| org.apache.unomi.opensearch.addresses=localhost:9200 | ||
| org.apache.unomi.opensearch.cluster.name=contextOpenSearch | ||
| org.apache.unomi.opensearch.sslEnable=false | ||
| org.apache.unomi.opensearch.username=admin | ||
| org.apache.unomi.opensearch.password=admin | ||
| ---- | ||
|
|
||
| 3. Restart Apache Unomi | ||
| + | ||
| [source,bash] | ||
| ---- | ||
| ./bin/stop | ||
| ./bin/start | ||
| ---- | ||
|
|
||
| == Troubleshooting | ||
|
|
||
| === Common Issues | ||
|
|
||
| 1. Connection Timeouts | ||
| * Increase the timeout settings in your configuration | ||
| * Check network connectivity between clusters | ||
|
|
||
| 2. Memory Issues | ||
| * Adjust JVM heap size for the migration tool | ||
| * Consider reducing batch sizes | ||
|
|
||
| 3. Missing Indices | ||
| * Verify index patterns in configuration | ||
| * Check source cluster health | ||
|
|
||
| === Monitoring Progress | ||
|
|
||
| The OpenSearch Replication Tool provides progress information during migration: | ||
|
|
||
| * Documents copied | ||
| * Time elapsed | ||
| * Current transfer rate | ||
| * Estimated completion time | ||
|
|
||
| == Best Practices | ||
|
|
||
| 1. *Testing* | ||
| * Always test the migration process in a non-production environment first | ||
| * Verify all Unomi features work with migrated data | ||
|
|
||
| 2. *Performance* | ||
| * Run migration during off-peak hours | ||
| * Monitor system resources during migration | ||
| * Use appropriate batch sizes based on document size | ||
|
|
||
| 3. *Backup* | ||
| * Create backups of your Elasticsearch indices before migration | ||
| * Keep source cluster running until verification is complete | ||
|
|
||
| 4. *Validation* | ||
| * Compare document counts between source and target | ||
| * Verify index mappings and settings | ||
| * Test Unomi functionality with migrated data | ||
|
|
||
| == Support | ||
|
|
||
| For additional support: | ||
|
|
||
| * OpenSearch Replication Tool: https://github.com/opensearch-project/opensearch-migrations | ||
| * Apache Unomi Community: https://unomi.apache.org/community.html | ||
| * OpenSearch Forum: https://forum.opensearch.org/ |