Deploy Openmetadata CI environment

[Ardiea]

Description/Context

Once I get a data-ci K8S cluster up and running, we need to look at creating an application stack(s) for OpenMetadata.

We need two actual stacks based on the docs:

• infrastructure.aws.opensearch.openmetadata.CI (I think this is already done or near done?)
• applications.openmetadata.CI
  • Logically divide this one into x parts
    • RDS stuff (MySQL/Maria I think?)
    • K8S Resources
    • Other stuff (IAM, networking, whatever comes up as you go)

RDS should have vault dynamic credentials configured like we normally do.
K8S Resources (helm charts and anything else) can be defined entirely with pulumi.

infrastructure.aws.eks.data.CI stack:

• will export a kube_config file that you will need for this creating k8s resources via pulumi.
• will create the openmetadata namespace for you.
• Will, eventually, be running a set of treafik pods in the operations namespace that will hook into the various "gateway api" resources we will need in addition to the openmetadata helm chart
• Will, eventually, be running the vault-secrets-operator that can be used to load the dynamic database creds into kubernetes Secrets that the app can pull into the pods.
• Will, eventually, be running exteranl-dns to automatically make openmetadata-ci.odl.mit.edu (or whatever).
• Will, eventually, be running cert-manager to go out to lets encrypt and get certificates for whatever URLs you create.

Many of those things are advanced topics, to start with focus on creating:

- RDS instace + Vault Auth stuff for it

• There may be firewall/SG challenges here, to start with just open the entire pod address space -> RDS instance.

• k8s resources with pulumi (refer to my branch / code that creates the cluster for examples of how to do this. It is very easy)

  • Just helm release to start, other stuff to follow.
  • You can create DB creds from the vault UI and load them in as Secrets manually to start, we can figure out the vault-secrets-operator stuff together later. I'm still struggling with it a bit myself.
  • I don't know how you did the open search stack, but the easiest thing might be to just connect to it over the internet to start with. The config would look a lot like the one for Learn. Then just load the username and pass into a Secret like you do with the database above.

• Once those are working, lets talk.

  • The best way to decide that the helm release has installed and is working is to check the pods and ensure they are all up and running without reboot cycling.

Other stuff:

• https://github.com/open-metadata/openmetadata-helm-charts
• https://github.com/open-metadata/openmetadata-helm-charts/blob/main/charts/openmetadata/values.yaml
• https://github.com/open-metadata/openmetadata-helm-charts?tab=readme-ov-file#quickstart
  • This has you loading dummy user+name passwords into Secrets, you'll do the same thing only not dummys, real ones. Make sure you don't check these yaml files into git.
• There are really two charts that need Release resources created.
  • openmetadata-dependencies
  • openmetadata
  • You can skip the dependencies? Maybe? Probably? You're smart, you'll figure it out. 👍 🎉 🌮

[feoh]

OMD CI opensearch is actually up and running:

    cluster       : {
        arn        : "arn:aws:es:us-east-1:610119931565:domain/opensearch-omdci"
        domain_id  : "610119931565/opensearch-omdci"
        domain_name: "opensearch-omdci"
        endpoint   : "vpc-opensearch-omdci-fizakvw3jvi575yxlvhdktlacq.us-east-1.es.amazonaws.com"
        urn        : "urn:pulumi:infrastructure.aws.opensearch.open_metadata.CI::ol-infrastructure-opensearch::aws:opensearch/domain:Domain::opensearch-v2-domain-cluster"
    }
    security_group: "sg-07078b7fd6d36aa5c"

Resources:
    + 8 created

[feoh]

Sooo close!

Was getting errors accessing the various API groups and realized I needed to add AWSEKSClusterAdministrator IAM policy to my IAM identity to make it work.

I'd mistakenly added AWSEKSAdministrator first, so we may need to do another spin down/up to get past this:

Diagnostics:
  kubernetes:helm.sh/v3:Release (open-metadata-CI-application-helm-release):
    error: 1 error occurred:
        * Helm release "open-metadata/open-metadata" failed to initialize completely. Use Helm CLI to investigate: failed to become available within allocated timeout. Error: Helm Release open-metadata/open-metadata: context deadline exceeded

  pulumi:pulumi:Stack (ol-infrastructure-open_metadata-application-applications.open_metadata.CI):
    error: update failed

Resources:
    ~ 1 updated
    41 unchanged

Duration: 5m20s