update metrics for rotate kubelet server certificate #122834

kannon92 · 2024-01-17T22:35:13Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

While drafting the retrospective KEP for RotateKubeletServerCertificate, we realize that the metrics are still in alpha stage.

This PR bumps the metrics to Beta. They have been in for a long time as alpha. Our goal is push this feature to GA.

Retrospective KEP is kubernetes/enhancements#4411.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Promote stability level for metrics associated with kubelet server certificate rotation (these are now **beta**).

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[KEP]: https://github.com/kubernetes/enhancements/issues/267

kannon92 · 2024-01-17T23:07:54Z

/cc @SergeyKanzhelev
/triage accepted
/priority important-longterm

I am working on promoting this feature to GA. I have a retrospective kep (from @SergeyKanzhelev!) and we indentified that these metrics should at least be promoted to beta.

I opened this up but not sure if the retrospective KEP should be merged first or not.

k8s-ci-robot · 2024-01-18T17:00:42Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: kannon92
Once this PR has been reviewed and has the lgtm label, please assign rainbowmango, smarterclayton for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-triage-robot · 2024-01-18T23:06:44Z

This PR may require stable metrics review.

Stable metrics are guaranteed to not change. Please review the documentation for the requirements and lifecycle of stable metrics and ensure that your metrics meet these guidelines.

sftim · 2024-01-19T01:14:48Z

Changelog suggestion

-Bump metrics for RotateKubeletServerCertificate to BETA
+Fixed stability level for metrics associated with kubelet certificate rotation (these are now **beta**).

If it's not a fix, change “Fixed” to “Promoted“.

kannon92 · 2024-01-19T14:21:12Z

/retest

kannon92 · 2024-01-19T14:21:45Z

CLA has an issue. Going to see if reopening will trigger it again.

That worked.

dgrisonnet · 2024-01-25T17:51:04Z

for sig-instrumentation review
/assign

tallclair · 2024-01-31T01:02:08Z

test/instrumentation/testdata/stable-metrics-list.yaml

+  subsystem: kubelet
+  help: Histogram of the number of seconds the previous certificate lived before being
+    rotated.
+  type: Histogram


Is histogram the right way to measure this? The choice seems a little odd to me.

I can't really answer yes or no to this. I mainly bumped this metrics which have been in alpha for a long while to beta.

We will at least need to have a consensus on https://github.com/kubernetes/kubernetes/pull/122834/files#r1472191704 if we want to bump this metric to BETA since once it is graduated it will be hard to change it in the future.

tallclair · 2024-01-31T01:06:27Z

test/instrumentation/testdata/stable-metrics-list.yaml

+  - 1.24416e+08
+- name: certificate_manager_server_ttl_seconds
+  subsystem: kubelet
+  help: Gauge of the shortest TTL (time-to-live) of the Kubelet's serving certificate.


What does shortest mean here? Does this mean that each time the certificate is rotated, this metric is updated to be min(previous metric value, ttl of new cert)?

tallclair · 2024-01-31T01:07:36Z

test/instrumentation/testdata/stable-metrics-list.yaml

+  - 1.24416e+08
+- name: certificate_manager_server_ttl_seconds
+  subsystem: kubelet
+  help: Gauge of the shortest TTL (time-to-live) of the Kubelet's serving certificate.


x509 certs don't actually have a TTL field. Does TTL here refer to the remaining TTL (time until expiry), or the total TTL (length of validity period)?

tallclair · 2024-01-31T01:08:02Z

test/instrumentation/testdata/stable-metrics-list.yaml

+  stabilityLevel: BETA
+- name: server_expiration_renew_errors
+  subsystem: kubelet
+  help: Counter of certificate renewal errors.


Suggested change

help: Counter of certificate renewal errors.

help: Counter of server certificate renewal errors.

tallclair · 2024-01-31T01:12:03Z

test/instrumentation/testdata/stable-metrics-list.yaml

+  subsystem: kubelet
+  help: Gauge of the shortest TTL (time-to-live) of the Kubelet's serving certificate.
+    The value is in seconds until certificate expiry (negative if already expired).
+    If serving certificate is invalid or unused, the value will be +INF.


What does unused mean here?

So this was autogenerated from the comments in kubelet.go for the metric.

We can discuss the commented metrics but I'm not sure if we should change this or not..

You shouldn't change the generated metadata, but you can always modify the metric. In this particular case, you could update the HELP text.

tallclair · 2024-01-31T01:12:43Z

test/instrumentation/testdata/stable-metrics-list.yaml

+    If serving certificate is invalid or unused, the value will be +INF.
+  type: Gauge
+  stabilityLevel: BETA
+- name: server_expiration_renew_errors


Should this be prepended with certificate_manager_ to match the other metrics?

tallclair · 2024-01-31T01:17:13Z

test/instrumentation/testdata/stable-metrics-list.yaml

+- name: certificate_manager_server_rotation_seconds
+  subsystem: kubelet
+  help: Histogram of the number of seconds the previous certificate lived before being
+    rotated.


This seems like a weird metric... it's measured as the time since it's validity start period to the time it was rotated:

kubernetes/staging/src/k8s.io/client-go/util/certificate/certificate_manager.go

Lines 595 to 597 in 7080b51

if old := m.updateCached(cert); old != nil && m.certificateRotation != nil {

m.certificateRotation.Observe(m.now().Sub(old.Leaf.NotBefore).Seconds())

}

This seems less useful than measuring the amount of valid time remaining (i.e. how close are we cutting it). It's also not measuring how long the cert was actually in use.

What is the purpose of this metric?

I agree, I also don't really understand how someone would use this information and take actions.

tallclair · 2024-01-31T01:17:30Z

/assign

kannon92 · 2024-02-06T19:17:37Z

test/instrumentation/testdata/stable-metrics-list.yaml

@@ -88,6 +88,35 @@
  help: The count of hidden metrics.


@tallclair all of this was autogenerated from the metrics. I guess beta metrics are actually documented so I'm not sure if I can/will change anything in this file. I can update the comments for the metrics but not sure about that.

kannon92 · 2024-02-06T19:18:55Z

ping @dgrisonnet

Curious on your thoughts for @tallclair! I didn't add the original comments for this metrics. I just wanted to promote the existing ones to beta but I'm open to making the comments more useful especially now that we will start documenting them more.

dgrisonnet · 2024-02-07T13:19:11Z

pkg/kubelet/certificate/kubelet.go

@@ -61,7 +61,7 @@ func NewKubeletServerCertificateManager(kubeClient clientset.Interface, kubeCfg
 			Subsystem:      metrics.KubeletSubsystem,
 			Name:           "server_expiration_renew_errors",


This is a counter so it needs to be suffixed by _total: https://prometheus.io/docs/practices/naming/

So I would suggest renaming the metric to server_expiration_renew_errors_total and highlight that change in the changelog.

dgrisonnet · 2024-02-07T13:21:55Z

test/instrumentation/testdata/stable-metrics-list.yaml

+- name: certificate_manager_server_rotation_seconds
+  subsystem: kubelet
+  help: Histogram of the number of seconds the previous certificate lived before being
+    rotated.


I agree, I also don't really understand how someone would use this information and take actions.

dgrisonnet · 2024-02-07T13:23:23Z

test/instrumentation/testdata/stable-metrics-list.yaml

+  subsystem: kubelet
+  help: Histogram of the number of seconds the previous certificate lived before being
+    rotated.
+  type: Histogram


We will at least need to have a consensus on https://github.com/kubernetes/kubernetes/pull/122834/files#r1472191704 if we want to bump this metric to BETA since once it is graduated it will be hard to change it in the future.

dgrisonnet · 2024-02-07T13:24:40Z

test/instrumentation/testdata/stable-metrics-list.yaml

+  subsystem: kubelet
+  help: Gauge of the shortest TTL (time-to-live) of the Kubelet's serving certificate.
+    The value is in seconds until certificate expiry (negative if already expired).
+    If serving certificate is invalid or unused, the value will be +INF.


You shouldn't change the generated metadata, but you can always modify the metric. In this particular case, you could update the HELP text.

tallclair · 2024-02-07T18:10:54Z

Just to clarify on the comments here, I commented on the auto-generated code since that's what was surfaced in this PR. The comments really apply to the metrics code, not the generated files here. I know you're just promoting what is already there, but as @dgrisonnet mentioned, promoting the metrics makes it harder to change them, so now is the time to evaluate the state of the metrics and think about what changes we want to make to them.

k8s-triage-robot · 2024-05-07T18:56:27Z

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle stale
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot · 2024-06-06T19:25:31Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle rotten
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot · 2024-07-06T19:48:43Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Reopen this PR with /reopen
Mark this PR as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot · 2024-07-06T19:48:48Z

@k8s-triage-robot: Closed this PR.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Reopen this PR with /reopen

Mark this PR as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot requested review from liggitt and mikedanese January 17, 2024 22:36

kannon92 mentioned this pull request Jan 17, 2024

Kubelet Server TLS Certificate Rotation kubernetes/enhancements#267

Open

k8s-ci-robot requested a review from SergeyKanzhelev January 17, 2024 23:07

update metrics for rotate kubelet server certificate

71e9bf7

kannon92 force-pushed the kubelet-certificate-metrics-bump branch from cc81559 to 71e9bf7 Compare January 18, 2024 16:59

kannon92 closed this Jan 19, 2024

kannon92 reopened this Jan 19, 2024

k8s-ci-robot assigned dgrisonnet Jan 25, 2024

tallclair reviewed Jan 31, 2024

View reviewed changes

k8s-ci-robot assigned tallclair Jan 31, 2024

kannon92 commented Feb 6, 2024

View reviewed changes

dgrisonnet reviewed Feb 7, 2024

View reviewed changes

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 7, 2024

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jun 6, 2024

kannon92 mentioned this pull request Jul 1, 2024

Graduate RotateKubeletServerCertificate #125811

Open

k8s-ci-robot closed this Jul 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update metrics for rotate kubelet server certificate #122834

update metrics for rotate kubelet server certificate #122834

kannon92 commented Jan 17, 2024 •

edited

Loading

kannon92 commented Jan 17, 2024

k8s-ci-robot commented Jan 18, 2024

k8s-triage-robot commented Jan 18, 2024

sftim commented Jan 19, 2024 •

edited

Loading

kannon92 commented Jan 19, 2024

kannon92 commented Jan 19, 2024 •

edited

Loading

dgrisonnet commented Jan 25, 2024

tallclair Jan 31, 2024

kannon92 Feb 6, 2024

dgrisonnet Feb 7, 2024

tallclair Jan 31, 2024

tallclair Jan 31, 2024

tallclair Jan 31, 2024

tallclair Jan 31, 2024

kannon92 Feb 6, 2024

dgrisonnet Feb 7, 2024

tallclair Jan 31, 2024

tallclair Jan 31, 2024

dgrisonnet Feb 7, 2024

tallclair commented Jan 31, 2024

kannon92 Feb 6, 2024

kannon92 commented Feb 6, 2024

dgrisonnet Feb 7, 2024

dgrisonnet Feb 7, 2024

dgrisonnet Feb 7, 2024

dgrisonnet Feb 7, 2024

tallclair commented Feb 7, 2024

k8s-triage-robot commented May 7, 2024

k8s-triage-robot commented Jun 6, 2024

k8s-triage-robot commented Jul 6, 2024

k8s-ci-robot commented Jul 6, 2024

	help: Counter of certificate renewal errors.
	help: Counter of server certificate renewal errors.

	if old := m.updateCached(cert); old != nil && m.certificateRotation != nil {
	m.certificateRotation.Observe(m.now().Sub(old.Leaf.NotBefore).Seconds())
	}

		@@ -61,7 +61,7 @@ func NewKubeletServerCertificateManager(kubeClient clientset.Interface, kubeCfg
		Subsystem: metrics.KubeletSubsystem,
		Name: "server_expiration_renew_errors",

update metrics for rotate kubelet server certificate #122834

update metrics for rotate kubelet server certificate #122834

Conversation

kannon92 commented Jan 17, 2024 • edited Loading

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

kannon92 commented Jan 17, 2024

k8s-ci-robot commented Jan 18, 2024

k8s-triage-robot commented Jan 18, 2024

sftim commented Jan 19, 2024 • edited Loading

kannon92 commented Jan 19, 2024

kannon92 commented Jan 19, 2024 • edited Loading

dgrisonnet commented Jan 25, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tallclair commented Jan 31, 2024

Choose a reason for hiding this comment

kannon92 commented Feb 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tallclair commented Feb 7, 2024

k8s-triage-robot commented May 7, 2024

k8s-triage-robot commented Jun 6, 2024

k8s-triage-robot commented Jul 6, 2024

k8s-ci-robot commented Jul 6, 2024

kannon92 commented Jan 17, 2024 •

edited

Loading

sftim commented Jan 19, 2024 •

edited

Loading

kannon92 commented Jan 19, 2024 •

edited

Loading