OpenTelemery Collector keeps crashing with OOMKilled status

[andrewgkew]

I have opentelemery collector setup to scrape 5 endpoints

skywalking metrics
kubernetes cadvisor
kube-state-metrics
stiod
envoy-stats

I found that the otel collector pod keeps crashing with status OOMKilled status, which is an out of memory issue.

So I figured either there is too much for it to collect or it needs more memory
So I tried the following:

1. Increase the replicas - I thought more pods to process the data should help
   OUTCOME: They all now crash with the same OOMKilled status

2. Increase their memory request - I increase this from 400Mi to 800Mi
   OUTCOME: Nothing changed there they all still crash with OOMKilled

Checking the node resources using k top nodes at the time the pods are crashing its at approx 60% so it has not over shot its memory on the actual machine

Question: Have you seen this issue before and/or any advice on how to handle this?

Do I have the collector setup with too many jobs and maybe should separate them?

I am using the following image on k8s cluster

image: otel/opentelemetry-collector:0.29.0

Thanks

[dashpole]

When the pod OOMs, you can look at the event to see which cgroup limit was hit. Requests don't matter at all for OOM; are you setting container limits on the collector container?

Unless you have implemented some form of target sharding (like we are working on in open-telemetry/prometheus-interoperability-spec#60), each replica of your collector is scraping all of the targets in your config. Adding more replicas won't decrease memory usage, unless you also split your scrape configs in some way.

Without dynamic target sharding, you can split by:

1. Manually split the config (e.g. run one collector for cAdvisor, one for KSM, etc)
2. Run the collector as a daemonset, and filter targets by node name. As an example for pods:

kubernetes_sd_configs:
- role: pod
   selectors:
   - role: pod
      field: spec.nodeName=$NODE_NAME

Both KSM and cAdvisor produce a lot of metrics. If there are any of them you don't need, dropping them with metric_relabel_configs (action == drop) can reduce memory usage significantly.

In my experience, the memory_limiter processor is a very effective tool for replacing OOM, which is a complete failure, with dropping metrics, which is a partial failure. I'd recommend using it even after you mitigate your problem so that when you run out of memory in the future, it just causes you to drop a few of them instead of falling over.

[andrewgkew]

@dashpole thanks for your help

There are some container limits yes

resources:
          limits:
            cpu: 1
            memory: 2Gi
          requests:
            cpu: 200m
            memory: 800Mi

I am not doing any form of sharing so what you say makes sense why each replica fails no matter how many I add.

I like the idea of the daemonset and node filter. Does it only accept node name can I not use labels instead?

Yes that is true I might see if I can remove some of the metrics from those 2 jobs.

I havent looked into the memory limiter processor I will take a look

Is the image version I am using ok as well?

[dashpole]

The image version should be fine...

The idea behind the daemonset is for each replica to only collect from the pods on the same node as it (and, I suppose, from the cAdvisor on that node as well). I forgot to add that $NODE_NAME is populated via the downward API like this:

env:
- name: NODE_NAME
   valueFrom:
   fieldRef:
     fieldPath: spec.nodeName

The important part is that for each replica the filter you are applying is different, because the NODE_NAME env var on each pod is different, such that each pod/cadvisor/etc is only scraped by a single collector. While you can put label selectors in the filter above, not sure of a way to use a different label selector for each collector.

[andrewgkew]

No problem thanks @dashpole I will give that a go

I am also going to include some additional proxyStatsMatcher.inclusionRegexps configurations on my Istio mesh installation which will reduce how much data is being pushed to OTEL as well

That combined with separating the jobs per node I am sure will help

I appreciate your help and speedy reply