Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OCPBUGS-19625: Multus per-node certificate request #2009

Merged
merged 1 commit into from
Sep 23, 2023

Conversation

dougbtv
Copy link
Member

@dougbtv dougbtv commented Sep 19, 2023

No description provided.

@dougbtv
Copy link
Member Author

dougbtv commented Sep 19, 2023

/hold

@openshift-ci openshift-ci bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Sep 19, 2023
@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 19, 2023
@s1061123
Copy link
Contributor

Hi

  • multus serviceaccunt no longer requried in per node cert because multus will query k8s api using new cert that is retrieved by per-node cert
  • Current ClusterRole should be applied to system:multus as follows:
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: multus-group
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: multus
subjects:
  - kind: Group
    name: system:multus
    apiGroup: rbac.authorization.k8s.io

@dougbtv dougbtv force-pushed the csr-request-update branch 6 times, most recently from 141349b to 8c17e36 Compare September 21, 2023 11:55
@dougbtv
Copy link
Member Author

dougbtv commented Sep 21, 2023

/retitle Multus per-node certificate request

@openshift-ci openshift-ci bot changed the title [wip] multus per node cert request Multus per-node certificate request Sep 21, 2023
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 21, 2023
@dougbtv dougbtv force-pushed the csr-request-update branch 2 times, most recently from 76b1988 to c579770 Compare September 21, 2023 19:08
@dougbtv
Copy link
Member Author

dougbtv commented Sep 21, 2023

/retitle OCPBUGS-19625: Multus per-node certificate request

@openshift-ci openshift-ci bot changed the title Multus per-node certificate request OCPBUGS-19625: Multus per-node certificate request Sep 21, 2023
@openshift-ci-robot openshift-ci-robot added jira/severity-critical Referenced Jira bug's severity is critical for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. labels Sep 21, 2023
@openshift-ci-robot
Copy link
Contributor

@dougbtv: This pull request references Jira Issue OCPBUGS-19625, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.15.0) matches configured target version for branch (4.15.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

No GitHub users were found matching the public email listed for the QA contact in Jira ([email protected]), skipping review request.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@dougbtv
Copy link
Member Author

dougbtv commented Sep 21, 2023

/hold cancel

@openshift-ci openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 21, 2023
@knobunc
Copy link
Contributor

knobunc commented Sep 22, 2023

/retest-required

1 similar comment
@zshi-redhat
Copy link
Contributor

/retest-required

valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: MULTUS_NODE_NAME
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI failure:

level=error msg=Cluster operator network Degraded is True with ApplyOperatorConfig: Error while updating operator configuration: could not apply (apps/v1, Kind=DaemonSet) openshift-multus/multus: failed to apply / update (apps/v1, Kind=DaemonSet) openshift-multus/multus: failed to create typed patch object (openshift-multus/multus; apps/v1, Kind=DaemonSet): .spec.template.spec.containers[name="kube-multus"].env: duplicate entries for key [name="MULTUS_NODE_NAME"]

https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_cluster-network-operator/2009/pull-ci-openshift-cluster-network-operator-master-e2e-gcp-ovn/1705053800420610048

It might be related to the fact that Dan already added this variable here: #2020

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, yep, I think it needs a rebase likely. good eye

roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: multus-proper
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call. Added conditions for RBAC + serviceaccount + serviceaccountname, as well as multus configuration to disable cert per node mode.

@@ -299,6 +313,9 @@ spec:
items:
- key: daemon-config.json
path: daemon-config.json
- name: host-run-multus-certs
hostPath:
path: /run/multus_certs
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be removed if a node reboots right?
I don't think we want to wipe the certs if they are still valid.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great observation. Turns out that we'll need a modification to ensure that we re-use it on the Multus end instead of re-request. This will certainly be an improvement, but we'll address it as a follow-on item.

[{
"commonNamePrefix":"system:multus",
"allowedPodAnnotations": ["k8s.v1.cni.cncf.io/network-status"]
}]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT: missing new line

@@ -121,6 +121,11 @@ data:
"logToStderr": true,
"logLevel": "verbose",
"binDir": "{{ .CNIBinDir }}",
"perNodeCertificate": {
Copy link
Contributor

@kyrtapz kyrtapz Sep 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like currently multus hard-codes cert-duration to 10min: https://github.com/k8snetworkplumbingwg/multus-cni/pull/1159/files#diff-562bab31c287044ff5fd9a5ba21041103225e129ca4373634b389356bdce0070R101

I think that the cert rotation shouldn't happen that often in a production environment.
In comparison, kubelet and ovn-kube use 24h.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Chatted with @knobunc and we're going to submit this as a follow on due to the dependent change in Multus. Great eye, though, this is necessary. 💯

@dougbtv dougbtv force-pushed the csr-request-update branch 5 times, most recently from 2c8bbb4 to 0fbefec Compare September 22, 2023 16:45
@jcaamano
Copy link
Contributor

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Sep 22, 2023
@abhat
Copy link
Contributor

abhat commented Sep 22, 2023

/test ci/prow/e2e-aws-sdn-multi
/test ci/prow/e2e-gcp-sdn
/test ci/prow/e2e-hypershift-ovn

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Sep 22, 2023

@abhat: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

  • /test 4.14-upgrade-from-stable-4.13-images
  • /test e2e-aws-ovn-network-migration
  • /test e2e-aws-ovn-windows
  • /test e2e-aws-sdn-multi
  • /test e2e-aws-sdn-network-migration-rollback
  • /test e2e-aws-sdn-network-reverse-migration
  • /test e2e-gcp-ovn
  • /test e2e-gcp-sdn
  • /test e2e-hypershift-ovn
  • /test e2e-metal-ipi-ovn-ipv6
  • /test e2e-vsphere-ovn-windows
  • /test images
  • /test lint
  • /test unit
  • /test verify

The following commands are available to trigger optional jobs:

  • /test 4.14-upgrade-from-stable-4.13-e2e-aws-ovn-upgrade
  • /test 4.14-upgrade-from-stable-4.13-e2e-azure-ovn-upgrade
  • /test 4.14-upgrade-from-stable-4.13-e2e-gcp-ovn-upgrade
  • /test e2e-aws-hypershift-ovn-kubevirt
  • /test e2e-aws-ovn-local-to-shared-gateway-mode-migration
  • /test e2e-aws-ovn-serial
  • /test e2e-aws-ovn-shared-to-local-gateway-mode-migration-periodic
  • /test e2e-aws-ovn-single-node
  • /test e2e-aws-sdn-upgrade
  • /test e2e-azure-ovn
  • /test e2e-azure-ovn-dualstack
  • /test e2e-azure-ovn-manual-oidc
  • /test e2e-gcp-ovn-upgrade
  • /test e2e-metal-ipi-ovn-ipv6-ipsec
  • /test e2e-network-mtu-migration-ovn-ipv4
  • /test e2e-network-mtu-migration-ovn-ipv6
  • /test e2e-network-mtu-migration-sdn-ipv4
  • /test e2e-openstack-kuryr
  • /test e2e-openstack-ovn
  • /test e2e-openstack-sdn
  • /test e2e-ovn-hybrid-step-registry
  • /test e2e-ovn-ipsec-step-registry
  • /test e2e-ovn-step-registry
  • /test e2e-vsphere-ovn
  • /test e2e-vsphere-ovn-dualstack
  • /test qe-perfscale-aws-ovn-medium-cluster-density
  • /test qe-perfscale-aws-ovn-medium-node-density-cni
  • /test qe-perfscale-aws-ovn-small-cluster-density
  • /test qe-perfscale-aws-ovn-small-node-density-cni

Use /test all to run the following jobs that were automatically triggered:

  • pull-ci-openshift-cluster-network-operator-master-4.14-upgrade-from-stable-4.13-images
  • pull-ci-openshift-cluster-network-operator-master-e2e-aws-hypershift-ovn-kubevirt
  • pull-ci-openshift-cluster-network-operator-master-e2e-aws-ovn-serial
  • pull-ci-openshift-cluster-network-operator-master-e2e-aws-ovn-shared-to-local-gateway-mode-migration-periodic
  • pull-ci-openshift-cluster-network-operator-master-e2e-aws-ovn-single-node
  • pull-ci-openshift-cluster-network-operator-master-e2e-aws-ovn-windows
  • pull-ci-openshift-cluster-network-operator-master-e2e-aws-sdn-multi
  • pull-ci-openshift-cluster-network-operator-master-e2e-aws-sdn-network-migration-rollback
  • pull-ci-openshift-cluster-network-operator-master-e2e-aws-sdn-network-reverse-migration
  • pull-ci-openshift-cluster-network-operator-master-e2e-aws-sdn-upgrade
  • pull-ci-openshift-cluster-network-operator-master-e2e-azure-ovn
  • pull-ci-openshift-cluster-network-operator-master-e2e-gcp-ovn
  • pull-ci-openshift-cluster-network-operator-master-e2e-gcp-ovn-upgrade
  • pull-ci-openshift-cluster-network-operator-master-e2e-gcp-sdn
  • pull-ci-openshift-cluster-network-operator-master-e2e-hypershift-ovn
  • pull-ci-openshift-cluster-network-operator-master-e2e-metal-ipi-ovn-ipv6
  • pull-ci-openshift-cluster-network-operator-master-e2e-metal-ipi-ovn-ipv6-ipsec
  • pull-ci-openshift-cluster-network-operator-master-e2e-network-mtu-migration-ovn-ipv4
  • pull-ci-openshift-cluster-network-operator-master-e2e-network-mtu-migration-ovn-ipv6
  • pull-ci-openshift-cluster-network-operator-master-e2e-network-mtu-migration-sdn-ipv4
  • pull-ci-openshift-cluster-network-operator-master-e2e-openstack-ovn
  • pull-ci-openshift-cluster-network-operator-master-e2e-openstack-sdn
  • pull-ci-openshift-cluster-network-operator-master-e2e-ovn-hybrid-step-registry
  • pull-ci-openshift-cluster-network-operator-master-e2e-ovn-ipsec-step-registry
  • pull-ci-openshift-cluster-network-operator-master-e2e-ovn-step-registry
  • pull-ci-openshift-cluster-network-operator-master-e2e-vsphere-ovn
  • pull-ci-openshift-cluster-network-operator-master-e2e-vsphere-ovn-dualstack
  • pull-ci-openshift-cluster-network-operator-master-e2e-vsphere-ovn-windows
  • pull-ci-openshift-cluster-network-operator-master-images
  • pull-ci-openshift-cluster-network-operator-master-lint
  • pull-ci-openshift-cluster-network-operator-master-unit
  • pull-ci-openshift-cluster-network-operator-master-verify

In response to this:

/test ci/prow/e2e-aws-sdn-multi
/test ci/prow/e2e-gcp-sdn
/test ci/prow/e2e-hypershift-ovn

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@abhat
Copy link
Contributor

abhat commented Sep 22, 2023

/test e2e-aws-sdn-multi
/test e2e-gcp-sdn
/test e2e-hypershift-ovn

@openshift-ci-robot
Copy link
Contributor

/retest-required

Remaining retests: 0 against base HEAD 90cc246 and 2 for PR HEAD 0fbefec in total

@openshift-ci openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Sep 22, 2023
@knobunc
Copy link
Contributor

knobunc commented Sep 22, 2023

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Sep 22, 2023
This introduces configurations for multus per-node certification.
@openshift-ci openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Sep 22, 2023
@knobunc
Copy link
Contributor

knobunc commented Sep 22, 2023

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Sep 22, 2023
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Sep 22, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dougbtv, jcaamano, knobunc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [dougbtv,jcaamano,knobunc]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@knobunc
Copy link
Contributor

knobunc commented Sep 22, 2023

/retest-required

1 similar comment
@s1061123
Copy link
Contributor

/retest-required

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Sep 23, 2023

@dougbtv: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-vsphere-ovn 5a97ce5 link false /test e2e-vsphere-ovn
ci/prow/e2e-network-mtu-migration-ovn-ipv4 5a97ce5 link false /test e2e-network-mtu-migration-ovn-ipv4
ci/prow/e2e-network-mtu-migration-sdn-ipv4 5a97ce5 link false /test e2e-network-mtu-migration-sdn-ipv4
ci/prow/e2e-openstack-sdn 5a97ce5 link false /test e2e-openstack-sdn
ci/prow/e2e-network-mtu-migration-ovn-ipv6 5a97ce5 link false /test e2e-network-mtu-migration-ovn-ipv6
ci/prow/e2e-vsphere-ovn-dualstack 5a97ce5 link false /test e2e-vsphere-ovn-dualstack
ci/prow/e2e-aws-ovn-serial 5a97ce5 link false /test e2e-aws-ovn-serial
ci/prow/e2e-metal-ipi-ovn-ipv6-ipsec 5a97ce5 link false /test e2e-metal-ipi-ovn-ipv6-ipsec
ci/prow/e2e-ovn-ipsec-step-registry 5a97ce5 link false /test e2e-ovn-ipsec-step-registry

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@openshift-merge-robot openshift-merge-robot merged commit 860bbce into openshift:master Sep 23, 2023
@openshift-ci-robot
Copy link
Contributor

@dougbtv: Jira Issue OCPBUGS-19625: Some pull requests linked via external trackers have merged:

The following pull requests linked via external trackers have not merged:

These pull request must merge or be unlinked from the Jira bug in order for it to move to the next state. Once unlinked, request a bug refresh with /jira refresh.

Jira Issue OCPBUGS-19625 has not been moved to the MODIFIED state.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@dougbtv
Copy link
Member Author

dougbtv commented Sep 23, 2023

/retest-required

@openshift-merge-robot
Copy link
Contributor

Fix included in accepted release 4.15.0-0.nightly-2023-09-27-073353

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/severity-critical Referenced Jira bug's severity is critical for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants