✨ Deploy DataHub into DPAT Development Cluster

[julialawrence]

User Story

As a member of the Data Labs team, I would like to experiment with the features of DataHub data catalogue.

Value / Purpose

Members of the data labs team are exploring potential alternatives to Open Metadata, which has issues around how it maps catalogue entities to cloud entities creating confusion

Useful Contacts

@julialawrence

User Types

Data Platform Labs

Proposal

Use Acryl helm charts to deploy, via terraform, a close-to-productionised version of DataHub in our Modernisation Platform Apps and Tools cluster, alongside with OpenMetadata - same cluster.

In order to facilitate load testing, a separate RDS postgres instance should also be created, instead of using a cluster-hosted one.

Authentication should be managed via AzureAD.

Additional Information

Chart code here: https://github.com/acryldata/datahub-helm
Instructions here: https://datahubproject.io/docs/deploy/kubernetes/

The deployment process isn't straight-forward as there are quite a few configuration options and prerequisites to be deployed as well.

Definition of Done

• RDS created
• Prerequisites deployed
• DataHub deployed in development DPAT cluster
• Access provided to the labs team.
• OpenSearch deployed
• Helm deployed via code

[julialawrence]

Noi further work will be carried out until the productionisation is agreed.