option for memory use limits or limits for number of keys

[grimnight]

Can you add option for memory use limits or limits for number of keys?

[yichengq]

Is this related to #2371 ?

[grimnight]

This is not related to #2371 . I would just like to be able to limit memory use.

[kelseyhightower]

@grimnight Yes this is not related.

A couple of questions come to mind for a feature like this. When memory usage limits are reached how should etcd handle new writes? Expiring TTLs -- should we let them expire and risk losing data?

[bprashanth]

+9000 to main memory limits, it's a common concept in most databases and I was puzzled that it wasn't configurable in etcd. I'd prefer having etcd swap to disk when it hits the limit, if that's not possible, flip to read-only and spam the logs with memory limit reached for each write. Just my 2c.

[klausenbusk]

Any update on this? Currently etcd2 use 90M or 18% of my memory (512MB Digital Ocean node)

[xiang90]

@klausenbusk What is the problem with 90M memory usage?

[klausenbusk]

@xiang90 It a large portion of the memory, when the nodes is only 512MB. Before I restarted etcd it was 136MB, I was testing something.

[xiang90]

@klausenbusk If you want to limit the memory usage, you can always limit the number of keys yourself; we can set a hard limit too to avoid you adding more keys into etcd. However etcd will keep ALL data in memory.

[klausenbusk]

I only have < 30 keys, but I refresh some of them very often. The 3 db nodes alone do around ~ 140 "refresh" every minute, then I also have some other node, but around ~ 200 "refresh" per minute.

[xiang90]

@kalabiyau etcd keeps internal history events for watch/replication etc.. Also etcd2 is not memory efficient and it is improved with our new v3 backend.