Pulumi RDS, Consul, k8s and supporting infra for OpenMetaData CI (#2593)

* feat: initial stack configuration for opensearch - OpenMetaData.

* Fix: use data platform consul for openmetadata opensearch.

* #2680

feat: Initial skeleton for MIT OL Open Metadata application Pulumi code.

* feat: Add Pulumi code for rds, consul and supporting configs

* fix: Remove unneeded IAM role, allow ingress from k8s subnet.

* Very WIP

* feat: Add Open Metadata helm chart, values, plumbing for db, es

* added namespace validation and reorg'd things a little. Added a vault auth backend role.

* Created a component resource for managing the vault secrets operator overhead. Added dynamic secrets to the openmetadata chart and made some experimental changes for the values.

* Bump pgsql engine major version to 16

* Fix reference to db name used in secret name. Also silence ludicrous TODO linting rules

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix ResourceOpts to ResourceOptions

* Fixed bugs to get a clean preview.

* Fixed bugs to get a clean preview.

* Tweaks to get a clean install.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: Use MariaDB not pg

* Revert "fix: Use MariaDB not pg"

This reverts commit bf7f3ae.

* Fixed configuration of env vars for db creds. Set up to use postgresql. Removed two version configs that snuck back in during rebase.

* Added additional grants to the application user.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fixed opensearch, fixed some formatting.

* feat: add traefik gateway ConfigGroup - broken

* Adding keycloak configurations for open-metadata.

* Fix errant reference to cluster name

* Added missing http httproute, other tweaks to gateway stuff.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Added missing http httproute, other tweaks to gateway stuff.

* Updated

* Fix Callback URL

* Extra dependencies for vault+rds instances.

* Fixing the database role creation process and defined some more dependencies between resources. Set delete_before_replace on all k8s resources.

---------

Co-authored-by: Ardiea <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

src/bridge/lib/versions.py

@@ -6,6 +6,7 @@
 CONSUL_VERSION = "1.14.0"
 DAGSTER_VERSION = "1.0.17"
 KEYCLOAK_VERSION = "24.0"
+OPEN_METADATA_VERSION = "1.5.5"
 OVS_VERSION = "v0.65.1-3-g2630021"
 REDASH_VERSION = "9d273e4"
 TRAEFIK_VERSION = "2.10.3"

src/ol_infrastructure/applications/open_metadata/Pulumi.applications.open_metadata.CI.yaml

@@ -0,0 +1,11 @@
+---
+secretsprovider: awskms://alias/infrastructure-secrets-ci
+encryptedkey: AQICAHjnbqe9AmEW1Js10nySybyuAG7Fb5E9EHUgkmqFDv7PxQHNccb0Qf344tYSpNK++I71AAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMqCuKG4WhkI4eNWUvAgEQgDtsdnkazeO8dfabHEekMYh4JgcxnwOf4B9F7zUoAkOMmqIQ8xQHmru06mRLnpK7v8QqW2zNACm6C2gUuA==
+config:
+  aws:region: us-east-1
+  consul:address: https://consul-data-ci.odl.mit.edu
+  open_metadata:domain: "open-metadata-ci.ol.mit.edu"
+  open_metadata:db_password:
+    secure: v1:N26+x5PLL2Teu2wf:8NVzfMjmFDD+rOZOMmWaSpSP6zbFIg+DbuWsiFOan4UdUuhHOG4wZDsjCJLWFafRKCXlRJq/CvBI5zw=
+  vault:address: https://vault-ci.odl.mit.edu
+  vault_server:env_namespace: operations.ci

src/ol_infrastructure/applications/open_metadata/Pulumi.yaml

@@ -0,0 +1,6 @@
+---
+name: ol-infrastructure-open_metadata-application
+runtime: python
+description: MIT OL Open Metadata Service
+backend:
+  url: s3://mitol-pulumi-state/