Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs(elastic-for-graph): Add migrating from neo4j to elastic instructions #2826

Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion datahub-kubernetes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ title: "Deploying with Kubernetes"
## Introduction
[This directory](https://github.com/linkedin/datahub/tree/master/datahub-kubernetes) provides
the Kubernetes [Helm](https://helm.sh/) charts for deploying [Datahub](https://github.com/linkedin/datahub/tree/master/datahub-kubernetes/datahub) and it's [dependencies](https://github.com/linkedin/datahub/tree/master/datahub-kubernetes/prerequisites)
(Elasticsearch, Neo4j, MySQL, and Kafka) on a Kubernetes cluster.
(Elasticsearch, optionally Neo4j, MySQL, and Kafka) on a Kubernetes cluster.

## Setup
1. Set up a kubernetes cluster
Expand Down
1 change: 1 addition & 0 deletions docs-website/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,7 @@ module.exports = {
"docs/advanced/high-cardinality",
"docs/how/scsi-onboarding-guide",
"docs/advanced/no-code-upgrade",
"docs/how/migrating-graph-service-implementation",
// WIP "docs/advanced/backfilling",
// WIP "docs/advanced/derived-aspects",
// WIP "docs/advanced/entity-hierarchy",
Expand Down
44 changes: 44 additions & 0 deletions docs/how/migrating-graph-service-implementation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# Migrate Graph Service Implementation to Elasticsearch

We currently support either Elasticsearch or Neo4j as backend implementations for the graph service. We recommend
Elasticsearch for those looking for a lighter deployment or do not want to manage a Neo4j database.
If you started using Neo4j as your graph service backend, here is how you can migrate to Elasticsearch.

## Docker-compose

If you are running your instance through docker locally, you will want to spin up your Datahub instance with
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also add the command to do this using datahub cli?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated

elasticsearch as the backend manually:

```aidl
docker-compose \
-f quickstart/docker-compose-without-neo4j.quickstart.yml \
pull && \
docker-compose -p datahub \
-f quickstart/docker-compose-without-neo4j.quickstart.yml \
up
```

Next, run the following command from root to rebuild your graph index.

```
./docker/datahub-upgrade/datahub-upgrade.sh -u RestoreIndices
```

After this command completes, you should be migrated. Open up the DataHub UI and verify your relationships are
visible.

After you confirm the migration is successful, you must remove your neo4j volume by running

```aidl
docker volume rm datahub_neo4jdata
```

This prevents your DataHub instance from coming up in neo4j mode in the future.

## Helm

First, follow the [deployment helm guide](../../datahub-kubernetes/README.md#components) to set up your helm deployment
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you link to the line in the quickstart-without-neo4j.values.yaml that sets graph_service to elasticsearc?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated

to use elasticsearch as your backend. Then, follow the [restore-indices helm guide](./restore-indices.md) to re-build
your indexes.

Once the job completes, your data will be migrated.