Enforce update order for OVN for Ctlplane/EDPM #792

dprince · 2024-05-07T18:48:06Z

Enforce update order for OVN for Ctlplane/EDPM
Jira: OSPRH-6732

dprince · 2024-05-07T18:48:23Z

/hold

softwarefactory-project-zuul · 2024-05-07T20:16:23Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/8e0ea450916246cd80dd8146d3b20e6d

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 26m 44s
❌ podified-multinode-edpm-deployment-crc FAILURE in 1h 08m 29s
❌ cifmw-crc-podified-edpm-baremetal FAILURE in 1h 08m 36s
❌ openstack-operator-tempest-multinode FAILURE in 1h 06m 35s

pkg/openstack/dataplane.go

controllers/core/openstackversion_controller.go

pkg/openstack/dataplane.go

controllers/core/openstackversion_controller.go

pkg/openstack/keystone.go

pkg/openstack/version.go

controllers/core/openstackcontrolplane_controller.go

pkg/openstack/dataplane.go

controllers/core/openstackversion_controller.go

softwarefactory-project-zuul · 2024-05-15T18:42:54Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/1683d7919bce48a88c1732103ab81f82

❌ openstack-k8s-operators-content-provider FAILURE in 11m 16s
⚠️ podified-multinode-edpm-deployment-crc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ openstack-operator-tempest-multinode SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider

softwarefactory-project-zuul · 2024-05-23T14:54:43Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/21aabfd5a3c940a49daa0d26f131d5c3

❌ openstack-k8s-operators-content-provider FAILURE in 10m 52s
⚠️ podified-multinode-edpm-deployment-crc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ openstack-operator-tempest-multinode SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider

dprince · 2024-05-23T17:27:35Z

recheck

softwarefactory-project-zuul · 2024-05-23T19:44:02Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/b8c58d7c412e4c1388e0bb7edcdaf75e

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 50m 37s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 17m 44s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 15m 37s
❌ openstack-operator-tempest-multinode RETRY_LIMIT in 24m 06s

dprince · 2024-05-28T01:16:49Z

recheck

tests/functional/base_test.go

softwarefactory-project-zuul · 2024-06-04T21:44:40Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/662941fc411642439586560bc4a94706

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 08m 08s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 23m 34s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 24m 21s
❌ openstack-operator-tempest-multinode FAILURE in 1h 52m 11s

dprince · 2024-06-05T00:24:07Z

recheck

softwarefactory-project-zuul · 2024-06-05T02:22:52Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/a0359f9137894b62b3af503929e991b8

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 57m 44s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 22m 46s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 20m 15s
❌ openstack-operator-tempest-multinode FAILURE in 1h 42m 24s

tests/functional/base_test.go

softwarefactory-project-zuul · 2024-06-08T02:09:58Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/7aa36ac7e4c4487c9334f98a072cbaaf

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 01m 10s
❌ podified-multinode-edpm-deployment-crc FAILURE in 1h 39m 23s
❌ cifmw-crc-podified-edpm-baremetal FAILURE in 1h 33m 43s
❌ openstack-operator-tempest-multinode FAILURE in 1h 43m 48s

Jira: OSPRH-6732

slawqo

looks good for me, the only thing I am not sure if I understand correctly is how all those changes in various {service}.go files are related to the PR description :)

dprince · 2024-06-11T15:02:33Z

looks good for me, the only thing I am not sure if I understand correctly is how all those changes in various {service}.go files are related to the PR description :)

in order to ensure update order we need to make sure services have been deployed (ready state, and observed generation checks).

gibizer

Looks OK to me.

dprince · 2024-06-11T22:16:48Z

recheck

booxter · 2024-06-11T22:32:02Z

I haven't tried to run it, but looking at the code, the order of updates looks correct.

I couldn't figure out from just reading the code how we guarantee that only ovn-controller (and not e.g. nova-compute) is updated on DP, before CP reconcileNormal is triggered. Will it be managed using a separate Deployment that would run a single ovn DP service? (I assume updating nova-compute before CP nova services is a problem.) Perhaps someone could ELI5 to me. (Thank you.)

bshephar · 2024-06-12T06:47:52Z

I haven't tried to run it, but looking at the code, the order of updates looks correct.

I couldn't figure out from just reading the code how we guarantee that only ovn-controller (and not e.g. nova-compute) is updated on DP, before CP reconcileNormal is triggered. Will it be managed using a separate Deployment that would run a single ovn DP service? (I assume updating nova-compute before CP nova services is a problem.) Perhaps someone could ELI5 to me. (Thank you.)

Also trying to follow this. So we set the condition here:
https://github.com/openstack-k8s-operators/openstack-operator/pull/792/files#diff-32500fc60d27debdcd1f64468b83c0a318d79fa55591b2e17a5cb935d9fde650R248

But that just prevents the controller from continuing the update until the condition has been satisfied. It seems that the actual update of OVN on the Dataplane would need to be a manual process as described here:

https://github.com/openstack-k8s-operators/dataplane-operator/blob/main/docs/assemblies/proc_updating-the-data-plane-ovn.adoc

So this would happen first and then pause until the condition get satisfied. To satisfy the condition, the image deployed would need to match the image the update is expecting as determined by:
https://github.com/openstack-k8s-operators/openstack-operator/pull/792/files#diff-308714db370a47145837acae5ff60352d9f352513007441b8c694f2c45c1031dR38-R53

Then the update can continue.

At least that's my 10 minute read on what's happening here. The answer is that the user will create the deployment limited to the OVN service like:

apiVersion: dataplane.openstack.org/v1beta1
kind: OpenStackDataPlaneDeployment
metadata:
  name: edpm-deployment-ipam-update
spec:
  nodeSets:
    - openstack-edpm-ipam
    - <nodeSet_name>
    - ...
    - <nodeSet_name>
  servicesOverride:
    - ovn

dprince · 2024-06-12T11:30:46Z

Dataplane updates are manual for GA. There are some Jira's filed (https://issues.redhat.com/browse/OSPRH-6421) which might help us fully streamline the minor update workflow.

dprince · 2024-06-12T11:33:22Z

I do think we could also validate based on conditions set on the OpenStackVersion resource that when Dataplane resources get executed we are in the correct state. So for example if we need to just execute an OVN playbook we could have a crude validation on that. The adminstrator could always override this with Ansible, but I think a simple check like this could help us further guard the workflow in the future.

booxter · 2024-06-12T13:14:23Z

Thank you @bshephar @dprince this (rolling in Deployment just for ovn service) makes sense. It's a bit more leg work for a user but the main point is we have a plan.

stuggi

/lgtm

openshift-ci · 2024-06-12T13:20:21Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dprince, stuggi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [dprince,stuggi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

There is no good reason to have this setting set to anything but `false`. If set to `true`, this will render OVN controllers broken until northd on control plane is fully upgraded. Since [1], we guarantee that ovn controllers are always updated before northd. See more details in [2]. [1] openstack-k8s-operators/openstack-operator#792 [2] openstack-k8s-operators/edpm-ansible#651 Depends-On: openstack-k8s-operators/edpm-ansible#651

openshift-ci bot requested review from frenzyfriday and lewisdenny May 7, 2024 18:48

openshift-ci bot added the approved label May 7, 2024

openshift-ci bot added the do-not-merge/hold label May 7, 2024

lewisdenny removed their request for review May 8, 2024 00:36

dprince force-pushed the ovn_update_order_edpm branch from 2df42b5 to f996cec Compare May 8, 2024 13:21

dprince requested review from fmount, stuggi, abays and gibizer May 8, 2024 13:46

abays reviewed May 8, 2024

View reviewed changes

pkg/openstack/dataplane.go Outdated Show resolved Hide resolved

dprince commented May 8, 2024

View reviewed changes

pkg/openstack/dataplane.go Outdated Show resolved Hide resolved

gibizer reviewed May 9, 2024

View reviewed changes

slagle reviewed May 9, 2024

View reviewed changes

pkg/openstack/dataplane.go Outdated Show resolved Hide resolved

pkg/openstack/dataplane.go Show resolved Hide resolved

controllers/core/openstackversion_controller.go Outdated Show resolved Hide resolved

controllers/core/openstackversion_controller.go Outdated Show resolved Hide resolved

dprince force-pushed the ovn_update_order_edpm branch 3 times, most recently from 26f8e9f to 3e0df91 Compare May 15, 2024 18:29

dprince force-pushed the ovn_update_order_edpm branch 2 times, most recently from cf31654 to c67650d Compare May 23, 2024 13:42

dprince removed the do-not-merge/hold label May 23, 2024

dprince force-pushed the ovn_update_order_edpm branch from c67650d to ba53816 Compare June 4, 2024 19:35

dprince commented Jun 4, 2024

View reviewed changes

tests/functional/base_test.go Outdated Show resolved Hide resolved

stuggi reviewed Jun 5, 2024

View reviewed changes

tests/functional/base_test.go Outdated Show resolved Hide resolved

dprince force-pushed the ovn_update_order_edpm branch from ba53816 to 74e6417 Compare June 8, 2024 00:07

Enforce update order for OVN for Ctlplane/EDPM

43153af

Jira: OSPRH-6732

dprince force-pushed the ovn_update_order_edpm branch from 74e6417 to 98815e3 Compare June 10, 2024 19:44

EDPM condition and update order checks

ae9065e

dprince force-pushed the ovn_update_order_edpm branch from 98815e3 to ae9065e Compare June 11, 2024 12:42

slawqo reviewed Jun 11, 2024

View reviewed changes

gibizer reviewed Jun 11, 2024

View reviewed changes

booxter mentioned this pull request Jun 11, 2024

Set ovn_match_northd_version to false by default openstack-k8s-operators/edpm-ansible#651

Merged

bshephar self-requested a review June 12, 2024 06:48

fao89 mentioned this pull request Jun 12, 2024

Move ctlplane tests to ctlplane dir #844

Closed

stuggi approved these changes Jun 12, 2024

View reviewed changes

openshift-ci bot assigned stuggi Jun 12, 2024

openshift-ci bot added the lgtm label Jun 12, 2024

openshift-merge-bot bot merged commit ee07591 into openstack-k8s-operators:main Jun 12, 2024
7 checks passed

booxter mentioned this pull request Jun 12, 2024

Don't retain ovn_match_northd_version from adopted host openstack-k8s-operators/os-diff#53

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enforce update order for OVN for Ctlplane/EDPM #792

Enforce update order for OVN for Ctlplane/EDPM #792

dprince commented May 7, 2024 •

edited by openshift-ci bot

Loading

dprince commented May 7, 2024

softwarefactory-project-zuul bot commented May 7, 2024

softwarefactory-project-zuul bot commented May 15, 2024

softwarefactory-project-zuul bot commented May 23, 2024

dprince commented May 23, 2024

softwarefactory-project-zuul bot commented May 23, 2024

dprince commented May 28, 2024

softwarefactory-project-zuul bot commented Jun 4, 2024

dprince commented Jun 5, 2024

softwarefactory-project-zuul bot commented Jun 5, 2024

softwarefactory-project-zuul bot commented Jun 8, 2024

slawqo left a comment

dprince commented Jun 11, 2024

gibizer left a comment

dprince commented Jun 11, 2024

booxter commented Jun 11, 2024

bshephar commented Jun 12, 2024 •

edited

Loading

dprince commented Jun 12, 2024

dprince commented Jun 12, 2024

booxter commented Jun 12, 2024

stuggi left a comment

openshift-ci bot commented Jun 12, 2024

Enforce update order for OVN for Ctlplane/EDPM #792

Enforce update order for OVN for Ctlplane/EDPM #792

Conversation

dprince commented May 7, 2024 • edited by openshift-ci bot Loading

dprince commented May 7, 2024

softwarefactory-project-zuul bot commented May 7, 2024

softwarefactory-project-zuul bot commented May 15, 2024

softwarefactory-project-zuul bot commented May 23, 2024

dprince commented May 23, 2024

softwarefactory-project-zuul bot commented May 23, 2024

dprince commented May 28, 2024

softwarefactory-project-zuul bot commented Jun 4, 2024

dprince commented Jun 5, 2024

softwarefactory-project-zuul bot commented Jun 5, 2024

softwarefactory-project-zuul bot commented Jun 8, 2024

slawqo left a comment

Choose a reason for hiding this comment

dprince commented Jun 11, 2024

gibizer left a comment

Choose a reason for hiding this comment

dprince commented Jun 11, 2024

booxter commented Jun 11, 2024

bshephar commented Jun 12, 2024 • edited Loading

dprince commented Jun 12, 2024

dprince commented Jun 12, 2024

booxter commented Jun 12, 2024

stuggi left a comment

Choose a reason for hiding this comment

openshift-ci bot commented Jun 12, 2024

dprince commented May 7, 2024 •

edited by openshift-ci bot

Loading

bshephar commented Jun 12, 2024 •

edited

Loading