Skip to content

Commit

Permalink
[receiver/kubeletstats] Add new CPU utilization metrics (#27276)
Browse files Browse the repository at this point in the history
**Description:** 
Adds new CPU utilization metrics with respect to pod/container CPU
limits and requests

**Link to tracking Issue:** <Issue number if applicable>
Closes
#24905

**Testing:** <Describe what testing was performed and which tests were
added.>
Added new unit tests and tested locally
  • Loading branch information
TylerHelmuth authored Oct 6, 2023
1 parent f9a1a52 commit a33e63a
Show file tree
Hide file tree
Showing 15 changed files with 476 additions and 35 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,15 @@ change_type: enhancement
component: kubeletstatsreceiver

# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
note: Add new metrics for representing pod and memory consumption of pods and containers as a percentage of the defined resource limits.
note: Adds new `k8s.pod.cpu_limit_utilization`, `k8s.pod.cpu_request_utilization`, `k8s.container.cpu_limit_utilization`, and `k8s.container.cpu_request_utilization` metrics that represent the ratio of cpu used vs set limits and requests.

# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
issues: [25835]
issues: [27276]

# (Optional) One or more lines of additional information to render under the primary note.
# These lines will be padded with 2 spaces and then inserted directly into the document.
# Use pipe (|) for multiline entries.
subtext: These metrics represent how much of your resource limits a container or pod is consuming.
subtext:

# If your change doesn't affect end users or the exported elements of any package,
# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
Expand Down
2 changes: 1 addition & 1 deletion .chloggen/kubeletstats-memory-utilization.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ change_type: enhancement
component: kubeletstatsreceiver

# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
note: Adds new `k8s.pod.memory.utilization` and `container.memory.utilization` metrics that represent the ratio of memory used vs limits set.
note: Adds new `k8s.pod.memory_limit_utilization`, `k8s.pod.memory_request_utilization`, `k8s.container.memory_limit_utilization`, and `k8s.container.memory_request_utilization` metrics that represent the ratio of memory used vs set limits and requests.

# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
issues: [25894]
Expand Down
32 changes: 32 additions & 0 deletions receiver/kubeletstatsreceiver/documentation.md
Original file line number Diff line number Diff line change
Expand Up @@ -394,6 +394,22 @@ The time since the container started
| ---- | ----------- | ---------- | ----------------------- | --------- |
| s | Sum | Int | Cumulative | true |
### k8s.container.cpu_limit_utilization
Container cpu utilization as a ratio of the container's limits
| Unit | Metric Type | Value Type |
| ---- | ----------- | ---------- |
| 1 | Gauge | Double |
### k8s.container.cpu_request_utilization
Container cpu utilization as a ratio of the container's requests
| Unit | Metric Type | Value Type |
| ---- | ----------- | ---------- |
| 1 | Gauge | Double |
### k8s.container.memory_limit_utilization
Container memory utilization as a ratio of the container's limits
Expand All @@ -418,6 +434,22 @@ The time since the node started
| ---- | ----------- | ---------- | ----------------------- | --------- |
| s | Sum | Int | Cumulative | true |
### k8s.pod.cpu_limit_utilization
Pod cpu utilization as a ratio of the pod's total container limits. If any container is missing a limit the metric is not emitted.
| Unit | Metric Type | Value Type |
| ---- | ----------- | ---------- |
| 1 | Gauge | Double |
### k8s.pod.cpu_request_utilization
Pod cpu utilization as a ratio of the pod's total container requests. If any container is missing a request the metric is not emitted.
| Unit | Metric Type | Value Type |
| ---- | ----------- | ---------- |
| 1 | Gauge | Double |
### k8s.pod.memory_limit_utilization
Pod memory utilization as a ratio of the pod's total container limits. If any container is missing a limit the metric is not emitted.
Expand Down
9 changes: 5 additions & 4 deletions receiver/kubeletstatsreceiver/internal/kubelet/accumulator.go
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ func (a *metricDataAccumulator) nodeStats(s stats.NodeStats) {

currentTime := pcommon.NewTimestampFromTime(a.time)
addUptimeMetric(a.mbs.NodeMetricsBuilder, metadata.NodeUptimeMetrics.Uptime, s.StartTime, currentTime)
addCPUMetrics(a.mbs.NodeMetricsBuilder, metadata.NodeCPUMetrics, s.CPU, currentTime)
addCPUMetrics(a.mbs.NodeMetricsBuilder, metadata.NodeCPUMetrics, s.CPU, currentTime, resources{})
addMemoryMetrics(a.mbs.NodeMetricsBuilder, metadata.NodeMemoryMetrics, s.Memory, currentTime, resources{})
addFilesystemMetrics(a.mbs.NodeMetricsBuilder, metadata.NodeFilesystemMetrics, s.Fs, currentTime)
addNetworkMetrics(a.mbs.NodeMetricsBuilder, metadata.NodeNetworkMetrics, s.Network, currentTime)
Expand All @@ -76,7 +76,7 @@ func (a *metricDataAccumulator) podStats(s stats.PodStats) {

currentTime := pcommon.NewTimestampFromTime(a.time)
addUptimeMetric(a.mbs.PodMetricsBuilder, metadata.PodUptimeMetrics.Uptime, s.StartTime, currentTime)
addCPUMetrics(a.mbs.PodMetricsBuilder, metadata.PodCPUMetrics, s.CPU, currentTime)
addCPUMetrics(a.mbs.PodMetricsBuilder, metadata.PodCPUMetrics, s.CPU, currentTime, a.metadata.podResources[s.PodRef.UID])
addMemoryMetrics(a.mbs.PodMetricsBuilder, metadata.PodMemoryMetrics, s.Memory, currentTime, a.metadata.podResources[s.PodRef.UID])
addFilesystemMetrics(a.mbs.PodMetricsBuilder, metadata.PodFilesystemMetrics, s.EphemeralStorage, currentTime)
addNetworkMetrics(a.mbs.PodMetricsBuilder, metadata.PodNetworkMetrics, s.Network, currentTime)
Expand Down Expand Up @@ -108,9 +108,10 @@ func (a *metricDataAccumulator) containerStats(sPod stats.PodStats, s stats.Cont
}

currentTime := pcommon.NewTimestampFromTime(a.time)
resourceKey := sPod.PodRef.UID + s.Name
addUptimeMetric(a.mbs.ContainerMetricsBuilder, metadata.ContainerUptimeMetrics.Uptime, s.StartTime, currentTime)
addCPUMetrics(a.mbs.ContainerMetricsBuilder, metadata.ContainerCPUMetrics, s.CPU, currentTime)
addMemoryMetrics(a.mbs.ContainerMetricsBuilder, metadata.ContainerMemoryMetrics, s.Memory, currentTime, a.metadata.containerResources[sPod.PodRef.UID+s.Name])
addCPUMetrics(a.mbs.ContainerMetricsBuilder, metadata.ContainerCPUMetrics, s.CPU, currentTime, a.metadata.containerResources[resourceKey])
addMemoryMetrics(a.mbs.ContainerMetricsBuilder, metadata.ContainerMemoryMetrics, s.Memory, currentTime, a.metadata.containerResources[resourceKey])
addFilesystemMetrics(a.mbs.ContainerMetricsBuilder, metadata.ContainerFilesystemMetrics, s.Rootfs, currentTime)

a.m = append(a.m, a.mbs.ContainerMetricsBuilder.Emit(
Expand Down
15 changes: 11 additions & 4 deletions receiver/kubeletstatsreceiver/internal/kubelet/cpu.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,20 +10,27 @@ import (
"github.com/open-telemetry/opentelemetry-collector-contrib/receiver/kubeletstatsreceiver/internal/metadata"
)

func addCPUMetrics(mb *metadata.MetricsBuilder, cpuMetrics metadata.CPUMetrics, s *stats.CPUStats, currentTime pcommon.Timestamp) {
func addCPUMetrics(mb *metadata.MetricsBuilder, cpuMetrics metadata.CPUMetrics, s *stats.CPUStats, currentTime pcommon.Timestamp, r resources) {
if s == nil {
return
}
addCPUUsageMetric(mb, cpuMetrics.Utilization, s, currentTime)
addCPUUsageMetric(mb, cpuMetrics, s, currentTime, r)
addCPUTimeMetric(mb, cpuMetrics.Time, s, currentTime)
}

func addCPUUsageMetric(mb *metadata.MetricsBuilder, recordDataPoint metadata.RecordDoubleDataPointFunc, s *stats.CPUStats, currentTime pcommon.Timestamp) {
func addCPUUsageMetric(mb *metadata.MetricsBuilder, cpuMetrics metadata.CPUMetrics, s *stats.CPUStats, currentTime pcommon.Timestamp, r resources) {
if s.UsageNanoCores == nil {
return
}
value := float64(*s.UsageNanoCores) / 1_000_000_000
recordDataPoint(mb, currentTime, value)
cpuMetrics.Utilization(mb, currentTime, value)

if r.cpuLimit > 0 {
cpuMetrics.LimitUtilization(mb, currentTime, value/r.cpuLimit)
}
if r.cpuRequest > 0 {
cpuMetrics.RequestUtilization(mb, currentTime, value/r.cpuRequest)
}
}

func addCPUTimeMetric(mb *metadata.MetricsBuilder, recordDataPoint metadata.RecordDoubleDataPointFunc, s *stats.CPUStats, currentTime pcommon.Timestamp) {
Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit a33e63a

Please sign in to comment.