[tlse] TLS database connection #331

stuggi · 2024-03-07T17:12:54Z

The my.cnf file gets added to the secret holding the service configs. The content of my.cnf is centrally managed in the mariadb-operator and retrieved calling db.GetDatabaseClientConfig(tlsCfg)

Depends-On: openstack-k8s-operators/mariadb-operator#190
Depends-On: openstack-k8s-operators/mariadb-operator#191

Jira: OSPRH-4547

softwarefactory-project-zuul · 2024-03-07T20:11:01Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/1f54dd75ccae4fb8a111ac316509f770

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 11m 54s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 07m 01s
❌ telemetry-operator-multinode-autoscaling-tempest FAILURE in 1h 32m 25s

jlarriba · 2024-03-08T08:12:45Z

There is an error in the operator with the current changes while trying to run aodh:

2024-03-07T19:56:50Z	INFO	Controllers.Autoscaling	Reconciling Service 'aodh'	{"controller": "autoscaling", "controllerGroup": "telemetry.openstack.org", "controllerKind": "Autoscaling", "Autoscaling": {"name":"autoscaling","namespace":"openstack"}, "namespace": "openstack", "name": "autoscaling", "reconcileID": "13e65351-d3f3-425d-8578-52506d697264"}
2024-03-07T19:56:50Z	ERROR	Reconciler error	{"controller": "autoscaling", "controllerGroup": "telemetry.openstack.org", "controllerKind": "Autoscaling", "Autoscaling": {"name":"autoscaling","namespace":"openstack"}, "namespace": "openstack", "name": "autoscaling", "reconcileID": "13e65351-d3f3-425d-8578-52506d697264", "error": "Failed to get aodh database openstack  *v1beta1.Autoscaling openstack/autoscaling: MariaDBDatabase.mariadb.openstack.org \"aodh\" not found"}

jlarriba · 2024-03-08T08:21:41Z

controllers/autoscaling_controller.go

-	customData := map[string]string{common.CustomServiceConfigFileName: instance.Spec.Aodh.CustomServiceConfig}
+
+	// the aodh controller currently creates the db with the user
+	db, err := mariadbv1.GetDatabaseByName(ctx, h, instance.Spec.Aodh.DatabaseUser)


I suspect that this should be instance.Spec.Aodh.DatabaseName

this was intentional as DatabaseUser is also used in https://github.com/openstack-k8s-operators/telemetry-operator/blob/main/controllers/aodh_controller.go#L159 . will have a look in a bit. I guess the issue is that the Database is created in a different controller.

I think we are not consistent on how the database is called, I have submitted some changes to use DatabaseUser as the database name everywhere

@stuggi The split between autoscaling_controller.go and aodh_controller.go doesn't mean anything, its one controller split into 2 files. The reason is, that historically the autoscaling controller was a lot bigger (it included parts of the current metric storage controller).

After discussing with @vyzigold, we realized that the databasename should be fixed, and the issue is to call it as DatabaseUser. This has been fixed now and everywhere it is referred to by the constant autoscaling.DatabaseName

stuggi · 2024-03-08T11:02:51Z

I missed to relocate the DB creation. we need to create the DB before the config gets rendered

softwarefactory-project-zuul · 2024-03-08T11:26:00Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/5f5d47b08af847a88991665fb7224e7e

❌ openstack-k8s-operators-content-provider FAILURE in 22m 13s
⚠️ podified-multinode-edpm-deployment-crc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ telemetry-operator-multinode-autoscaling-tempest SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider

stuggi · 2024-03-08T11:33:16Z

recheck

softwarefactory-project-zuul · 2024-03-08T13:34:18Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/5ff5bda50b6d4e07b835e27207980586

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 59m 30s
❌ podified-multinode-edpm-deployment-crc FAILURE in 1h 26m 49s
❌ telemetry-operator-multinode-autoscaling-tempest FAILURE in 1h 33m 11s

jlarriba · 2024-03-08T13:55:12Z

There is something wrong with reconciliation, it never seems to end:

2024-03-08T13:25:58Z	INFO	Controllers.Autoscaling	Reconciling Service 'aodh'	{"controller": "autoscaling", "controllerGroup": "telemetry.openstack.org", "controllerKind": "Autoscaling", "Autoscaling": {"name":"autoscaling","namespace":"openstack"}, "namespace": "openstack", "name": "autoscaling", "reconcileID": "f86dd085-5b6e-43ee-b3c8-a71d029200be"}
2024-03-08T13:25:58Z	INFO	Controllers.Autoscaling	Database object for MariaDBDatabase aodh does not have a MariaDBAccount CR name configured. Assuming legacy use of the API, will use the same name for the MariaDBAccount.	{"controller": "autoscaling", "controllerGroup": "telemetry.openstack.org", "controllerKind": "Autoscaling", "Autoscaling": {"name":"autoscaling","namespace":"openstack"}, "namespace": "openstack", "name": "autoscaling", "reconcileID": "f86dd085-5b6e-43ee-b3c8-a71d029200be"}
2024-03-08T13:25:58Z	INFO	Controllers.Autoscaling	Applied new databasehostname openstack.openstack.svc to MariaDBDatabase aodh	{"controller": "autoscaling", "controllerGroup": "telemetry.openstack.org", "controllerKind": "Autoscaling", "Autoscaling": {"name":"autoscaling","namespace":"openstack"}, "namespace": "openstack", "name": "autoscaling", "reconcileID": "f86dd085-5b6e-43ee-b3c8-a71d029200be"}
2024-03-08T13:25:58Z	INFO	Controllers.Autoscaling	Waiting for MariaDBDatabase aodh to be fully reconciled	{"controller": "autoscaling", "controllerGroup": "telemetry.openstack.org", "controllerKind": "Autoscaling", "Autoscaling": {"name":"autoscaling","namespace":"openstack"}, "namespace": "openstack", "name": "autoscaling", "reconcileID": "f86dd085-5b6e-43ee-b3c8-a71d029200be", "ObjectType": "*v1beta1.MariaDBDatabase", "ObjectNamespace": "openstack", "ObjectName": "aodh"}

stuggi · 2024-03-08T14:11:45Z

There is something wrong with reconciliation, it never seems to end:

2024-03-08T13:25:58Z	INFO	Controllers.Autoscaling	Reconciling Service 'aodh'	{"controller": "autoscaling", "controllerGroup": "telemetry.openstack.org", "controllerKind": "Autoscaling", "Autoscaling": {"name":"autoscaling","namespace":"openstack"}, "namespace": "openstack", "name": "autoscaling", "reconcileID": "f86dd085-5b6e-43ee-b3c8-a71d029200be"}
2024-03-08T13:25:58Z	INFO	Controllers.Autoscaling	Database object for MariaDBDatabase aodh does not have a MariaDBAccount CR name configured. Assuming legacy use of the API, will use the same name for the MariaDBAccount.	{"controller": "autoscaling", "controllerGroup": "telemetry.openstack.org", "controllerKind": "Autoscaling", "Autoscaling": {"name":"autoscaling","namespace":"openstack"}, "namespace": "openstack", "name": "autoscaling", "reconcileID": "f86dd085-5b6e-43ee-b3c8-a71d029200be"}
2024-03-08T13:25:58Z	INFO	Controllers.Autoscaling	Applied new databasehostname openstack.openstack.svc to MariaDBDatabase aodh	{"controller": "autoscaling", "controllerGroup": "telemetry.openstack.org", "controllerKind": "Autoscaling", "Autoscaling": {"name":"autoscaling","namespace":"openstack"}, "namespace": "openstack", "name": "autoscaling", "reconcileID": "f86dd085-5b6e-43ee-b3c8-a71d029200be"}
2024-03-08T13:25:58Z	INFO	Controllers.Autoscaling	Waiting for MariaDBDatabase aodh to be fully reconciled	{"controller": "autoscaling", "controllerGroup": "telemetry.openstack.org", "controllerKind": "Autoscaling", "Autoscaling": {"name":"autoscaling","namespace":"openstack"}, "namespace": "openstack", "name": "autoscaling", "reconcileID": "f86dd085-5b6e-43ee-b3c8-a71d029200be", "ObjectType": "*v1beta1.MariaDBDatabase", "ObjectNamespace": "openstack", "ObjectName": "aodh"}

the dbName had the wrong label. changed it in my last update. lets see

softwarefactory-project-zuul · 2024-03-08T15:45:45Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/285636c1a12549e08066d375c181073b

❌ openstack-k8s-operators-content-provider RETRY_LIMIT in 31m 11s
⚠️ podified-multinode-edpm-deployment-crc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ telemetry-operator-multinode-autoscaling-tempest SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider

stuggi · 2024-03-08T16:03:21Z

recheck

softwarefactory-project-zuul · 2024-03-08T17:40:23Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/382b384dfb114029a25750cf4bf7e2b9

❌ openstack-k8s-operators-content-provider TIMED_OUT in 31m 17s
⚠️ podified-multinode-edpm-deployment-crc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ telemetry-operator-multinode-autoscaling-tempest SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider

jlarriba · 2024-03-11T07:51:05Z

Yesterday a PR was submitted to move the database to the new "DBAccounts" API: #333.

@stuggi I suggest to wait until that is merged and rebase this on top of that, WDYT?

stuggi · 2024-03-11T10:06:02Z

Yesterday a PR was submitted to move the database to the new "DBAccounts" API: #333.

@stuggi I suggest to wait until that is merged and rebase this on top of that, WDYT?

yes sounds good

jlarriba · 2024-03-12T08:18:17Z

Yesterday a PR was submitted to move the database to the new "DBAccounts" API: #333.
@stuggi I suggest to wait until that is merged and rebase this on top of that, WDYT?

yes sounds good

@stuggi #333 was merged, could you please rebase on top of it?

jlarriba · 2024-03-12T10:07:32Z

/unhold

softwarefactory-project-zuul · 2024-03-12T11:53:00Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/31cb6ddfefb4439aa8b5a19d2442d4e7

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 56m 52s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 04m 27s
❌ telemetry-operator-multinode-autoscaling-tempest FAILURE in 1h 33m 02s

The my.cnf file gets added to the secret holding the service configs. The content of my.cnf is centrally managed in the mariadb-operator and retrieved calling db.GetDatabaseClientConfig(tlsCfg) Depends-On: openstack-k8s-operators/mariadb-operator#190 Depends-On: openstack-k8s-operators/mariadb-operator#191 Jira: OSPRH-4547

softwarefactory-project-zuul · 2024-03-12T16:43:31Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/e2b696addf0041edb8b19b750498622d

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 29m 49s
❌ podified-multinode-edpm-deployment-crc RETRY_LIMIT in 59m 01s
✔️ telemetry-operator-multinode-autoscaling-tempest SUCCESS in 1h 01m 51s

jlarriba · 2024-03-12T16:44:12Z

recheck

softwarefactory-project-zuul · 2024-03-12T17:05:38Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/78b9731f24f2401c8aedb06fddb22baf

❌ openstack-k8s-operators-content-provider FAILURE in 19m 55s
⚠️ podified-multinode-edpm-deployment-crc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ telemetry-operator-multinode-autoscaling-tempest SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider

stuggi · 2024-03-12T17:11:17Z

recheck

stuggi · 2024-03-13T06:42:42Z

recheck

stuggi · 2024-03-13T14:53:00Z

    case.go:364: failed in step 2-deploy
    case.go:366: Internal error occurred: failed calling webhook "mautoscaling.kb.io": failed to call webhook: Post "https://telemetry-operator-controller-manager-service.openstack-operators.svc:443/mutate-telemetry-openstack-org-v1beta1-autoscaling?timeout=10s": no endpoints available for service "telemetry-operator-controller-manager-service"

stuggi · 2024-03-13T14:53:16Z

/test telemetry-operator-build-deploy-kuttl

openshift-ci · 2024-03-14T10:29:30Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jlarriba, stuggi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [jlarriba]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci bot requested review from lewisdenny and vyzigold March 7, 2024 17:13

jlarriba approved these changes Mar 8, 2024

View reviewed changes

openshift-ci bot assigned jlarriba Mar 8, 2024

openshift-ci bot added lgtm approved labels Mar 8, 2024

jlarriba removed the lgtm label Mar 8, 2024

jlarriba reviewed Mar 8, 2024

View reviewed changes

jlarriba force-pushed the tls_db branch 2 times, most recently from c1998f6 to 58fbe55 Compare March 8, 2024 09:38

stuggi force-pushed the tls_db branch from cbd024b to 78fbdf4 Compare March 8, 2024 14:07

stuggi marked this pull request as draft March 11, 2024 10:06

openshift-ci bot added the do-not-merge/work-in-progress label Mar 11, 2024

openshift-merge-robot added the needs-rebase label Mar 12, 2024

stuggi force-pushed the tls_db branch from 78fbdf4 to ab26a59 Compare March 12, 2024 09:25

openshift-merge-robot removed the needs-rebase label Mar 12, 2024

stuggi force-pushed the tls_db branch from ab26a59 to 7a31d41 Compare March 12, 2024 15:11

stuggi removed the do-not-merge/work-in-progress label Mar 12, 2024

stuggi marked this pull request as ready for review March 12, 2024 16:01

openshift-ci bot requested review from csibbitt and jlarriba March 12, 2024 16:01

jlarriba approved these changes Mar 14, 2024

View reviewed changes

openshift-ci bot added the lgtm label Mar 14, 2024

openshift-merge-bot bot merged commit 5590312 into openstack-k8s-operators:main Mar 14, 2024
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[tlse] TLS database connection #331

[tlse] TLS database connection #331

stuggi commented Mar 7, 2024 •

edited by openshift-ci bot

Loading

softwarefactory-project-zuul bot commented Mar 7, 2024

jlarriba commented Mar 8, 2024

jlarriba Mar 8, 2024

stuggi Mar 8, 2024

jlarriba Mar 8, 2024

vyzigold Mar 8, 2024

jlarriba Mar 8, 2024

stuggi commented Mar 8, 2024

softwarefactory-project-zuul bot commented Mar 8, 2024

stuggi commented Mar 8, 2024

softwarefactory-project-zuul bot commented Mar 8, 2024

jlarriba commented Mar 8, 2024 •

edited

Loading

stuggi commented Mar 8, 2024

softwarefactory-project-zuul bot commented Mar 8, 2024

stuggi commented Mar 8, 2024

softwarefactory-project-zuul bot commented Mar 8, 2024

jlarriba commented Mar 11, 2024

stuggi commented Mar 11, 2024

jlarriba commented Mar 12, 2024

jlarriba commented Mar 12, 2024

softwarefactory-project-zuul bot commented Mar 12, 2024

softwarefactory-project-zuul bot commented Mar 12, 2024

jlarriba commented Mar 12, 2024

softwarefactory-project-zuul bot commented Mar 12, 2024

stuggi commented Mar 12, 2024

stuggi commented Mar 13, 2024

stuggi commented Mar 13, 2024

stuggi commented Mar 13, 2024

openshift-ci bot commented Mar 14, 2024

[tlse] TLS database connection #331

[tlse] TLS database connection #331

Conversation

stuggi commented Mar 7, 2024 • edited by openshift-ci bot Loading

softwarefactory-project-zuul bot commented Mar 7, 2024

jlarriba commented Mar 8, 2024

jlarriba Mar 8, 2024

Choose a reason for hiding this comment

stuggi Mar 8, 2024

Choose a reason for hiding this comment

jlarriba Mar 8, 2024

Choose a reason for hiding this comment

vyzigold Mar 8, 2024

Choose a reason for hiding this comment

jlarriba Mar 8, 2024

Choose a reason for hiding this comment

stuggi commented Mar 8, 2024

softwarefactory-project-zuul bot commented Mar 8, 2024

stuggi commented Mar 8, 2024

softwarefactory-project-zuul bot commented Mar 8, 2024

jlarriba commented Mar 8, 2024 • edited Loading

stuggi commented Mar 8, 2024

softwarefactory-project-zuul bot commented Mar 8, 2024

stuggi commented Mar 8, 2024

softwarefactory-project-zuul bot commented Mar 8, 2024

jlarriba commented Mar 11, 2024

stuggi commented Mar 11, 2024

jlarriba commented Mar 12, 2024

jlarriba commented Mar 12, 2024

softwarefactory-project-zuul bot commented Mar 12, 2024

softwarefactory-project-zuul bot commented Mar 12, 2024

jlarriba commented Mar 12, 2024

softwarefactory-project-zuul bot commented Mar 12, 2024

stuggi commented Mar 12, 2024

stuggi commented Mar 13, 2024

stuggi commented Mar 13, 2024

stuggi commented Mar 13, 2024

openshift-ci bot commented Mar 14, 2024

stuggi commented Mar 7, 2024 •

edited by openshift-ci bot

Loading

jlarriba commented Mar 8, 2024 •

edited

Loading