Resource limits for spark dependencies cronjob

[harshal-shah]

The spark dependencies cronjob does not have resources populated. It would be nice to be able to specify requests and limits for the pods created by this cronjob.
Right now the pods created by this cronjob do not have any resources defined and they consume a substantial amount of CPU / memory causing resource starvation on the node and all pods on it.

[antimack]

+1 to this, we have default requests and limits per namespace and spark dependencies OOM killed all the time

[harshal-shah]

@antimack as a workaround, you can add a LimitRange object to the namespace where jaeger is installed.

[antimack]

@harshal-shah Yeah, that’s what we do but we have default limitrange same for all the namespaces and we are trying to keep it low so spark dependencies failing to start. I just tweaked limitrange a bit but having the ability to specify resources in crd would be way much better :)

[harshal-shah]

Yes, Totally agree with that.

[jpkrohling]

@rubenvp8510 is this something you'd want to work on?

[harshal-shah]

We removed the limitrange and instead used the following helmfile hook to patch the cronjob

    hooks:
    - events: ["postsync"]
      showlogs: true
      command: "kubectl"
      args: ["patch", "cronjob", "jaeger-spark-dependencies", "-n", "tracing", "--type", "merge", "--patch", "{ \"spec\": { \"jobTemplate\": { \"spec\": { \"template\": { \"spec\": { \"containers\": [ { \"name\": \"jaeger-spark-dependencies\", \"image\": \"jaegertracing/spark-dependencies\", \"resources\": { \"requests\":{ \"cpu\": \"2000m\", \"memory\": \"4Gi\" }, \"limits\":{ \"cpu\": \"2500m\", \"memory\": \"6Gi\" } } } ] } } } } } }"]

[harshal-shah]

Above patch gets overwritten by the operator

W0826 08:33:55.753589       1 reflector.go:256] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:93: watch of *v1beta1.CronJob ended with: too old resource version: 198112976 (198121115)