Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add cgroup RSS queries #143

Merged
merged 1 commit into from
Nov 13, 2024
Merged

Add cgroup RSS queries #143

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions cmd/config/metrics-aggregated.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -156,17 +156,37 @@
instant: true
captureStart: true

- query: sum ( container_memory_rss { id =~ "/system.slice|/system.slice/kubelet.service|/system.slice/ovs-vswitchd.service|/system.slice/crio.service|/kubepods.slice" } and on (node) kube_node_role{ role = "worker",role != "infra" } ) by ( id )
metricName: cgroupMemoryRSS-Workers
instant: true
captureStart: true

- query: sum ( container_cpu_usage_seconds_total { id =~ "/system.slice|/system.slice/kubelet.service|/system.slice/ovs-vswitchd.service|/system.slice/crio.service|/kubepods.slice" } and on (node) kube_node_role{ role = "master" } ) by ( id )
metricName: cgroupCPUSeconds-Masters
instant: true
captureStart: true

- query: sum ( container_memory_rss { id =~ "/system.slice|/system.slice/kubelet.service|/system.slice/ovs-vswitchd.service|/system.slice/crio.service|/kubepods.slice" } and on (node) kube_node_role{ role = "master" } ) by ( id )
metricName: cgroupMemoryRSS-Masters
instant: true
captureStart: true

- query: sum ( container_cpu_usage_seconds_total { id =~ "/system.slice|/system.slice/kubelet.service|/system.slice/ovs-vswitchd.service|/system.slice/crio.service|/kubepods.slice" } and on (node) kube_node_role{ role = "infra" } ) by ( id )
metricName: cgroupCPUSeconds-Infra
instant: true
captureStart: true

- query: sum ( container_memory_rss { id =~ "/system.slice|/system.slice/kubelet.service|/system.slice/ovs-vswitchd.service|/system.slice/crio.service|/kubepods.slice" } and on (node) kube_node_role{ role = "infra" } ) by ( id )
metricName: cgroupMemoryRSS-Infra
instant: true
captureStart: true

- query: sum( container_cpu_usage_seconds_total{container!~"POD|",namespace=~"openshift-.*"} ) by (namespace)
metricName: cgroupCPUSeconds-namespaces
instant: true
captureStart: true

- query: sum( container_memory_rss{container!~"POD|",namespace=~"openshift-.*"} ) by (namespace)
rsevilla87 marked this conversation as resolved.
Show resolved Hide resolved
metricName: cgroupMemoryRSS-namespaces
instant: true
captureStart: true
20 changes: 20 additions & 0 deletions cmd/config/metrics-egressip.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,17 +168,37 @@
instant: true
captureStart: true

- query: sum ( container_memory_rss { id =~ "/system.slice|/system.slice/kubelet.service|/system.slice/ovs-vswitchd.service|/system.slice/crio.service|/kubepods.slice" } and on (node) kube_node_role{ role = "worker",role != "infra" } ) by ( id )
metricName: cgroupMemoryRSS-Workers
instant: true
captureStart: true

- query: sum ( container_cpu_usage_seconds_total { id =~ "/system.slice|/system.slice/kubelet.service|/system.slice/ovs-vswitchd.service|/system.slice/crio.service|/kubepods.slice" } and on (node) kube_node_role{ role = "master" } ) by ( id )
metricName: cgroupCPUSeconds-Masters
instant: true
captureStart: true

- query: sum ( container_memory_rss { id =~ "/system.slice|/system.slice/kubelet.service|/system.slice/ovs-vswitchd.service|/system.slice/crio.service|/kubepods.slice" } and on (node) kube_node_role{ role = "master" } ) by ( id )
metricName: cgroupMemoryRSS-Masters
instant: true
captureStart: true

- query: sum ( container_cpu_usage_seconds_total { id =~ "/system.slice|/system.slice/kubelet.service|/system.slice/ovs-vswitchd.service|/system.slice/crio.service|/kubepods.slice" } and on (node) kube_node_role{ role = "infra" } ) by ( id )
metricName: cgroupCPUSeconds-Infra
instant: true
captureStart: true

- query: sum ( container_memory_rss { id =~ "/system.slice|/system.slice/kubelet.service|/system.slice/ovs-vswitchd.service|/system.slice/crio.service|/kubepods.slice" } and on (node) kube_node_role{ role = "infra" } ) by ( id )
metricName: cgroupMemoryRSS-Infra
instant: true
captureStart: true

- query: sum( container_cpu_usage_seconds_total{container!~"POD|",namespace=~"openshift-.*"} ) by (namespace)
metricName: cgroupCPUSeconds-namespaces
instant: true
captureStart: true

- query: sum( container_memory_rss{container!~"POD|",namespace=~"openshift-.*"} ) by (namespace)
metricName: cgroupMemoryRSS-namespaces
instant: true
captureStart: true
25 changes: 25 additions & 0 deletions cmd/config/metrics-report.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -300,6 +300,11 @@
metricName: max-memory-sum-workers
instant: true

- query: sum( (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) and on (instance) label_replace(kube_node_role{role="worker"}, "instance", "$1", "node", "(.+)") )
metricName: memory-sum-workers
instant: true
captureStart: true
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding this metric because it allows us to calculate the memory delta over the run, with start and end instant measurements. This can help us detect memory leaks over the course of a run.


- query: avg(avg_over_time(sum(irate(node_cpu_seconds_total{mode!="idle", mode!="steal"}[2m]) and on (instance) label_replace(kube_node_role{role="infra"}, "instance", "$1", "node", "(.+)")) by (instance)[{{.elapsed}}:]))
metricName: cpu-infra
instant: true
Expand Down Expand Up @@ -393,17 +398,37 @@
instant: true
captureStart: true

- query: sum ( container_memory_rss { id =~ "/system.slice|/system.slice/kubelet.service|/system.slice/ovs-vswitchd.service|/system.slice/crio.service|/kubepods.slice" } and on (node) kube_node_role{ role = "worker",role != "infra" } ) by ( id )
metricName: cgroupMemoryRSS-Workers
instant: true
captureStart: true

- query: sum(container_cpu_usage_seconds_total{id=~"/system.slice|/system.slice/kubelet.service|/system.slice/ovs-vswitchd.service|/system.slice/crio.service|/kubepods.slice"} and on (node) kube_node_role{role="master"}) by (id)
metricName: cgroupCPUSeconds-Masters
instant: true
captureStart: true

- query: sum ( container_memory_rss { id =~ "/system.slice|/system.slice/kubelet.service|/system.slice/ovs-vswitchd.service|/system.slice/crio.service|/kubepods.slice" } and on (node) kube_node_role{ role = "master" } ) by ( id )
metricName: cgroupMemoryRSS-Masters
instant: true
captureStart: true

- query: sum(container_cpu_usage_seconds_total{id=~"/system.slice|/system.slice/kubelet.service|/system.slice/ovs-vswitchd.service|/system.slice/crio.service|/kubepods.slice"} and on (node) kube_node_role{role="infra"}) by (id)
metricName: cgroupCPUSeconds-Infra
instant: true
captureStart: true

- query: sum ( container_memory_rss { id =~ "/system.slice|/system.slice/kubelet.service|/system.slice/ovs-vswitchd.service|/system.slice/crio.service|/kubepods.slice" } and on (node) kube_node_role{ role = "infra" } ) by ( id )
metricName: cgroupMemoryRSS-Infra
instant: true
captureStart: true

- query: sum(container_cpu_usage_seconds_total{container!~"POD|",namespace=~"openshift-.*"}) by (namespace)
metricName: cgroupCPUSeconds-namespaces
instant: true
captureStart: true

- query: sum( container_memory_rss{container!~"POD|",namespace=~"openshift-.*"} ) by (namespace)
metricName: cgroupMemoryRSS-namespaces
instant: true
captureStart: true
20 changes: 20 additions & 0 deletions cmd/config/metrics.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,17 +147,37 @@
instant: true
captureStart: true

- query: sum ( container_memory_rss { id =~ "/system.slice|/system.slice/kubelet.service|/system.slice/ovs-vswitchd.service|/system.slice/crio.service|/kubepods.slice" } and on (node) kube_node_role{ role = "worker",role != "infra" } ) by ( id )
metricName: cgroupMemoryRSS-Workers
instant: true
captureStart: true

- query: sum ( container_cpu_usage_seconds_total { id =~ "/system.slice|/system.slice/kubelet.service|/system.slice/ovs-vswitchd.service|/system.slice/crio.service|/kubepods.slice" } and on (node) kube_node_role{ role = "master" } ) by ( id )
metricName: cgroupCPUSeconds-Masters
instant: true
captureStart: true

- query: sum ( container_memory_rss { id =~ "/system.slice|/system.slice/kubelet.service|/system.slice/ovs-vswitchd.service|/system.slice/crio.service|/kubepods.slice" } and on (node) kube_node_role{ role = "master" } ) by ( id )
metricName: cgroupMemoryRSS-Masters
instant: true
captureStart: true

- query: sum ( container_cpu_usage_seconds_total { id =~ "/system.slice|/system.slice/kubelet.service|/system.slice/ovs-vswitchd.service|/system.slice/crio.service|/kubepods.slice" } and on (node) kube_node_role{ role = "infra" } ) by ( id )
metricName: cgroupCPUSeconds-Infra
instant: true
captureStart: true

- query: sum ( container_memory_rss { id =~ "/system.slice|/system.slice/kubelet.service|/system.slice/ovs-vswitchd.service|/system.slice/crio.service|/kubepods.slice" } and on (node) kube_node_role{ role = "infra" } ) by ( id )
metricName: cgroupMemoryRSS-Infra
instant: true
captureStart: true

- query: sum( container_cpu_usage_seconds_total{container!~"POD|",namespace=~"openshift-.*"} ) by (namespace)
metricName: cgroupCPUSeconds-namespaces
instant: true
captureStart: true

- query: sum( container_memory_rss{container!~"POD|",namespace=~"openshift-.*"} ) by (namespace)
metricName: cgroupMemoryRSS-namespaces
instant: true
captureStart: true
Loading