[0.10.0] CREATE CONTINUOUS QUERY causes lease conflicts in cluster

[e-dard]

When repeating this in a one-node cluster, I don't see these lease conflict issues.

When running a 3-node fully replicated (default settings) cluster, I get conflict responses in the logs of
the node where the CREATE CONTINUOUS QUERY was run, since another node (n2) has acquired the lease on the continuous query, and is running them.

// n1
> create database cqtest
> use cqtest
Using database cqtest
> insert cpu value=1
> CREATE CONTINUOUS QUERY cqmin ON cqtest BEGIN SELECT min(value) INTO cqtest."default".foo FROM cqtest."default".cpu GROUP BY time(10s) END

According to the logs, at this point n2 actually gets the lease for this CQ, and runs it every 10 seconds. However, in the mean time n1 is trying to get the lease, and apparently either succeeding (200 response), or getting a conflict (409 response).

I never see node n1 actually run the CQ according to the logs. n1 logs:

[query] 2016/02/08 15:11:55 CREATE CONTINUOUS QUERY cqmin ON cqtest BEGIN SELECT min(value) INTO cqtest."default".foo FROM cqtest."default".cpu GROUP BY time(10s) END
[meta] 2016/02/08 15:11:55 127.0.0.1 - - [08/Feb/2016:15:11:45 +0000] GET /?index=19 HTTP/1.1 200 353 - Go-http-client/1.1 44b439a0-ce76-11e5-8046-000000000000 10.032694794s
[meta] 2016/02/08 15:11:55 127.0.0.1 - - [08/Feb/2016:15:11:55 +0000] POST /execute HTTP/1.1 200 28 - Go-http-client/1.1 4aae7513-ce76-11e5-804b-000000000000 3.91892ms
[meta] 2016/02/08 15:11:55 127.0.0.1 - - [08/Feb/2016:15:11:45 +0000] GET /?index=19 HTTP/1.1 200 353 - Go-http-client/1.1 44b43b2b-ce76-11e5-8047-000000000000 10.032969374s
[meta] 2016/02/08 15:11:55 127.0.0.1 - - [08/Feb/2016:15:11:45 +0000] GET /?index=19 HTTP/1.1 200 353 - Go-http-client/1.1 44b437c0-ce76-11e5-8045-000000000000 10.033057196s
[http] 2016/02/08 15:11:55 ::1 - - [08/Feb/2016:15:11:55 +0000] GET /query?db=cqtest&epoch=ns&q=CREATE+CONTINUOUS+QUERY+cqmin+ON+cqtest+BEGIN+SELECT+min%28value%29+INTO+cqtest.%22default%22.foo+FROM+cqtest.%22default%22.cpu+GROUP+BY+time%2810s%29+END HTTP/1.1 200 40 - InfluxDBShell/0.9 4aae61bb-ce76-11e5-804a-000000000000 6.692855ms
[meta] 2016/02/08 15:11:55 127.0.0.1 - - [08/Feb/2016:15:11:55 +0000] GET /lease?name=continuous_querier&nodeid=4 HTTP/1.1 200 106 - Go-http-client/1.1 4aaf5d61-ce76-11e5-804f-000000000000 1.003967ms
[meta] 2016/02/08 15:11:55 127.0.0.1 - - [08/Feb/2016:15:11:55 +0000] GET /lease?name=continuous_querier&nodeid=5 HTTP/1.1 409 106 - Go-http-client/1.1 4ab20ba2-ce76-11e5-8050-000000000000 653.968µs
[meta] 2016/02/08 15:11:56 127.0.0.1 - - [08/Feb/2016:15:11:56 +0000] GET /lease?name=continuous_querier&nodeid=1 HTTP/1.1 409 106 - Go-http-client/1.1 4b2e2602-ce76-11e5-8051-000000000000 277.162µs
[meta] 2016/02/08 15:11:56 127.0.0.1 - - [08/Feb/2016:15:11:56 +0000] GET /lease?name=continuous_querier&nodeid=4 HTTP/1.1 200 106 - Go-http-client/1.1 4b48b052-ce76-11e5-8052-000000000000 1.158163ms
[meta] 2016/02/08 15:11:56 127.0.0.1 - - [08/Feb/2016:15:11:56 +0000] GET /lease?name=continuous_querier&nodeid=5 HTTP/1.1 409 106 - Go-http-client/1.1 4b4b99e9-ce76-11e5-8053-000000000000 223.563µs
[meta] 2016/02/08 15:11:57 127.0.0.1 - - [08/Feb/2016:15:11:57 +0000] GET /lease?name=continuous_querier&nodeid=1 HTTP/1.1 409 106 - Go-http-client/1.1 4bc73ab6-ce76-11e5-8054-000000000000 676.991µs
[meta] 2016/02/08 15:11:57 127.0.0.1 - - [08/Feb/2016:15:11:57 +0000] GET /lease?name=continuous_querier&nodeid=4 HTTP/1.1 200 106 - Go-http-client/1.1 4be1b4c6-ce76-11e5-8055-000000000000 367.806µs
[meta] 2016/02/08 15:11:57 127.0.0.1 - - [08/Feb/2016:15:11:57 +0000] GET /lease?name=continuous_querier&nodeid=5 HTTP/1.1 409 106 - Go-http-client/1.1 4be4d8a5-ce76-11e5-8056-000000000000 1.269661ms
[meta] 2016/02/08 15:11:58 127.0.0.1 - - [08/Feb/2016:15:11:58 +0000] GET /lease?name=continuous_querier&nodeid=1 HTTP/1.1 409 106 - Go-http-client/1.1 4c601a68-ce76-11e5-8057-000000000000 214.151µs
[meta] 2016/02/08 15:11:58 127.0.0.1 - - [08/Feb/2016:15:11:58 +0000] GET /lease?name=continuous_querier&nodeid=4 HTTP/1.1 200 106 - Go-http-client/1.1 4c7b3ccc-ce76-11e5-8058-000000000000 1.021503ms
[meta] 2016/02/08 15:11:58 127.0.0.1 - - [08/Feb/2016:15:11:58 +0000] GET /lease?name=continuous_querier&nodeid=5 HTTP/1.1 409 106 - Go-http-client/1.1 4c7e4703-ce76-11e5-8059-000000000000 249.477µs

n2 logs:

[run] 2016/02/08 15:11:29 Listening for signals
2016/02/08 15:11:29 updated node metaservers with: [localhost:8191 localhost:8291 localhost:8391]
[cluster] 2016/02/08 15:11:49 accept remote connection from [::1]:51314
[tsm1wal] 2016/02/08 15:11:49 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/02/08 15:11:49 tsm1 WAL writing to /Users/edd/Development/influx_cluster/n2/.influxdb/wal/_internal/monitor/1
[cluster] 2016/02/08 15:11:49 accept remote connection from [::1]:51316
[continuous_querier] 2016/02/08 15:12:00 executing continuous query cqmin (2016-02-08 15:11:50 +0000 GMT to 2016-02-08 15:12:00 +0000 GMT)
[query] 2016/02/08 15:12:00 SELECT min(value) INTO cqtest."default".foo FROM cqtest."default".cpu WHERE time >= '2016-02-08T15:11:50Z' AND time < '2016-02-08T15:12:00Z' GROUP BY time(10s)
[continuous_querier] 2016/02/08 15:12:10 executing continuous query cqmin (2016-02-08 15:12:00 +0000 GMT to 2016-02-08 15:12:10 +0000 GMT)
[query] 2016/02/08 15:12:10 SELECT min(value) INTO cqtest."default".foo FROM cqtest."default".cpu WHERE time >= '2016-02-08T15:12:00Z' AND time < '2016-02-08T15:12:10Z' GROUP BY time(10s)
[continuous_querier] 2016/02/08 15:12:20 executing continuous query cqmin (2016-02-08 15:12:10 +0000 GMT to 2016-02-08 15:12:20 +0000 GMT)
[query] 2016/02/08 15:12:20 SELECT min(value) INTO cqtest."default".foo FROM cqtest."default".cpu WHERE time >= '2016-02-08T15:12:10Z' AND time < '2016-02-08T15:12:20Z' GROUP BY time(10s)
[continuous_querier] 2016/02/08 15:12:30 executing continuous query cqmin (2016-02-08 15:12:20 +0000 GMT to 2016-02-08 15:12:30 +0000 GMT)
[query] 2016/02/08 15:12:30 SELECT min(value) INTO cqtest."default".foo FROM cqtest."default".cpu WHERE time >= '2016-02-08T15:12:20Z' AND time < '2016-02-08T15:12:30Z' GROUP BY time(10s)

Full logs attached.

n1.txt
n2.txt
n3.txt

[e-dard]

Closing as expected behaviour