Difference between metrics results node node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate AND pod:container_cpu_usage:sum

[ehvs]

Trying to determine what the difference is between the kube metrics as seen and leveraged in two different ways by OCP out-of-box custom records in Prometheus. If you simply sum each by pod and subtract from eachother you'll see the differences are very small, but on some higher CPU use pods like ETCD the difference it gets more skewed (greater diff).

My guess is that while both use "container_cpu_usage_seconds_total", one looks to sum/avg[5m] of active containers, whereas the later filters to a different metric that has the total CPU seconds for the entire life of the pod including any init container not still active?
{container!="POD",image!="",job="kubelet",metrics_path="/metrics/cadvisor"}
vs
{container="",pod!=""}

record: node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate
expr: sum
  by(cluster, namespace, pod, container) (rate(container_cpu_usage_seconds_total{container!="POD",image!="",job="kubelet",metrics_path="/metrics/cadvisor"}[5m]))
  * on(cluster, namespace, pod) group_left(node) topk by(cluster, namespace, pod)
  (1, max by(cluster, namespace, pod, node) (kube_pod_info{node!=""}))

record: pod:container_cpu_usage:sum
expr: sum
  by(pod, namespace) (rate(container_cpu_usage_seconds_total{container="",pod!=""}[5m]))

sum by (pod)(pod:container_cpu_usage:sum) 
- sum by (pod)(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate) < -0.001 

OR

sum by (pod)(pod:container_cpu_usage:sum) 
- sum by (pod)(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate) > 0.001 

What's further confusing, is taking the same raw metric sum/rate queries those records are based on and specifying them in your own query, same moment in time, I get different results.

sum by (pod)(
sum by (pod)(
sum
  by(pod, namespace) (rate(container_cpu_usage_seconds_total{container="",pod!=""}[5m]))
) 

sum by (pod) (
sum
  by(cluster, namespace, pod, container) (rate(container_cpu_usage_seconds_total{container!="POD",image!="",job="kubelet",metrics_path="/metrics/cadvisor"}[5m]))
  * on(cluster, namespace, pod) group_left(node) topk by(cluster, namespace, pod)
  (1, max by(cluster, namespace, pod, node) (kube_pod_info{node!=""}))
)
) < -0.001

[paulfantom]

Seems like a bug in label selectors we use for both queries. For example this shows more correct results:

sum by (namespace) (
sum by(cluster, namespace, pod, container) (
  rate(container_cpu_usage_seconds_total{container!="POD",image!="",job="kubelet",metrics_path="/metrics/cadvisor"}[5m])
) * on(cluster, namespace, pod) group_left(node) topk by(cluster, namespace, pod) (
  1, max by(cluster, namespace, pod, node) (kube_pod_info{node!=""})
)
)
-
sum by(namespace) (rate(container_cpu_usage_seconds_total{container!="POD",image!=""}[5m]))

I suggest to open a bug report in kubernetes-mixin project