Memory leak when "[ERR] agent: failed to sync remote state: rpc error"

[sgirones]

Consul agent stats

19047 root      20  0 41.1g 547m 6700 S  13  0.2  0:03.85 consul
[...]
19047 root      20  0 44.5g 3.9g 6828 S    4  1.5  0:43.20 consul

Consul logs:

[ERR] agent: failed to sync remote state: rpc error: rpc error: rpc error: rpc error: rpc error: rpc error: rpc error: rpc error: rpc error: rpc error: rpc error: rpc error: rpc error: rpc error: rpc error: rpc error: rpc error: rpc error: rpc error: rpc error: rpc error: rpc error: rpc error: rpc error: rpc error: rpc error: rpc error: rpc error: rpc error: rpc error: rpc error: rpc error: rpc error: rpc error: rpc error: rpc error [...]

Still we haven't had time to reproduce it, but maybe someone looking at the code can identify it. Deducing from the log, the problem can come from here:

consul/command/agent/local.go

Line 249 in 3d3c48b

for {

[ryanuber]

I've seen this once before as well, but wasn't able to reproduce it. Tagging as a bug, thanks!

[sgirones]

I can say it was solved once the consul data was cleaned. It had non-reachable IPs.

If I have time these days I will try to reproduce, but can't promise.

[armon]

Fixed by hashicorp/raft@a88bfa8