Etcd needs to be configured with auto-compaction option

[etiennedi]

Since we use etcd mostly for distributed locking, the default revisioning feature of etcd leads to the revision cache filling up and then etcd stopping it's work altogether

The manual clean up steps are described here:
https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/maintenance.md#space-quota

However, when bulk importing we'd have to do this manually every 4 hours or so .

There also seems to be an --auto-compaction option so this happens autmatically.

[etiennedi]

This very much depends on #50.

[idcrosby]

This can be set as an environment variable: ETCD_AUTO_COMPACTION_RETENTION (https://coreos.com/etcd/docs/latest/op-guide/configuration.html). The Helm chart I am currently testing allows us to set arbitrary env vars for Etcd Config.

[idcrosby]

@etiennedi do you have an idea how frequently (hours) the auto compaction should run? Is 4 reasonable?

[etiennedi]

@idcrosby 4 would probably be not frequent enough for a batch import scenario. I ran into the issue with too much revision history every 4-6 hours at my scale test, so I'd recommend something like 1 maybe to be on the safe side. However, after batch import is complete it could be much less frequent, so it probably makes sense to expose this parameter through our values.yml, so users can set it depending on the mode. (I was thinking of creating a few different values.yml as presets for different scenarios anyway. Maybe one for batch import, one for regular operation, etc.)

Does this mean the compaction runs once every <configured interval>? I was hoping it would run more or less all the time and only use the value set there as a retention period. This could become relevant because manually running the compaction after 4-6h took quite some time (~5 min). I'm wondering if the API would stay usable while such a big compaction is running. However, if it's constantly running in the background there might not be such a big compaction that has potential to block the API.

[idcrosby]

@etiennedi you're right, the setting simply enables auto-compaction and the value is the retention period for the key.

There is a second setting for how to run auto compaction (ETCD_AUTO_COMPACTION_MODE) which can be either periodic or revision based. The periodic compaction runs at 1/10th of the retention period.

[idcrosby]

#57