InfluxDB Consuming A Lot of Memory

[jinglejengel]

Rancher versions:
rancher/server: v1.6.0
rancher/agent :v0.5.4

Infrastructure Stack versions:
healthcheck: v0.3.1
ipsec: v0.1.0
network-services: v0.2.0
scheduler: v0.5.1
kubernetes (if applicable): v1.5.4-rancher1-4.1

Docker version: (docker version,docker info preferred)

Docker version 1.12.6, build 78d1802

Operating system and kernel: (cat /etc/os-release, uname -r preferred)

NAME="Ubuntu"
VERSION="16.04.2 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.2 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)

AWS Nodes

Setup details: (single node rancher vs. HA rancher, internal DB vs. external DB)

HA rancher with external DB

Environment Template: (Cattle/Kubernetes/Swarm/Mesos)

Kubernetes

Steps to Reproduce:

Let the cluster sit for a bit, launch some deployments and run some workloads.

Results:

InfluxDB will consume a bunch of memory

This particular cluster only has 121 containers running (that's including all the kubernetes/rancher goods), 4 AWS hosts to host those.

Talked with Ahmad in slack, they asked me to file an issue for it for discussion.

[aemneina]

I believe the add ons needs a version bump for influx db. It currently uses 1.1.1 and we should have 1.2.0+ as that contains the fixes here: influxdata/influxdb#7832

[alena1108]

@aemneina @Joeskyyy for influxdb we use images from the official kubernetes repo. They do build it and only use certain packages of the official influxdb repo (https://github.com/influxdata/influxdb). The version of those packages is v.0.12.2 (https://github.com/kubernetes/heapster/blob/master/Godeps/Godeps.json#L241). That eliminates a possibility of a memory leak that was introduced later, in 1.1.0 and 1.1.1.

So the bug can be as well in k8s heapster code base, and we need to file for https://github.com/kubernetes/kubernetes/issues. Not sure if there are any profiling tools enabled on the image so we could take the memory footprint to make it easier to debug for them.

[aemneina]

@alena1108 it looks like the addon starter is hard coded to use an older images for influx (1.1.1), the official k8s repo seems to be using v1.3.3 now. We should update the addon starter here. Heapster and grafana also have version bumps.

[deniseschannon]

@moelsayed Can you confirm what versions will be used in k8s 1.8? is the later version for influx compatible for the currently released k8s 1.7?

[moelsayed]

@deniseschannon the version we updated to in 1.8 is heapster-influxdb-amd64:v1.3.3. I am not 100% sure if its compatible with 1.7. We probably need to test that to confirm.

[moelsayed]

@deniseschannon I tested and the updated stacks we are using with k8s 1.8 work fine on the released k8s 1.7.

[moelsayed]

Updated versions of the addons are already merged, tested and working. The new versions should fix this issue:

heapster-grafana-amd64:v4.4.3
heapster-amd64:v1.4.0
heapster-influxdb-amd64:v1.3.3

rancher/kubernetes-package#97