Clustering 0.9.4 - already member of cluster

[derailed]

Hi

I am trying to cluster 2 nodes using 0.9.4 and can't seem to find the right incantation.
I have followed the clustering docs and check the issues but still no dice.

Here are my two attempts that both yield the same results ie

Skipping cluster join: already member of cluster: nodeId=1 raftEnabled=true peers=[influx2:8088]

ATTEMPT #1

Node1 config

[meta]
hostname = "influx1"
bind-address = ":8088"
...

Node2 config

[meta]
hostname = "influx2"
bind-address = ":8088"
...

NOTE influx1/influx2 are defined in etc/hosts

Start node1

influxd run -config influx1.toml

Start node2

influxd run -config influx2.toml -join influx1:8088

Results:
Skipping cluster join: already member of cluster: nodeId=1 raftEnabled=true peers=[influx2:8088]

ATTEMPT #2

Node1 config

[meta]
hostname = "influx1"
bind-address = ":8088"
peers = ["influx1:8088","influx2:8088"]
...

Node2 config

[meta]
hostname = "influx2"
bind-address = ":8088"
peers = ["influx1:8088","influx2:8088"]
...

Start node1

influxd run -config influx1.toml

Start node2

influxd run -config influx2.toml

Results:
Skipping cluster join: already member of cluster: nodeId=1 raftEnabled=true peers=[influx2:8088]

Also show servers on either influx1 or influx2 only show 1 node.

What am I missing?

Tx!!

[jwilder]

You probably have data in your node2 data dir so it is re-using that cluster info. Make sure that directory is empty before starting node 2.

[sandmanahoy]

Hey. I've actually been fighting the same issue for the past, like, 2 days. I can confirm that I do not have any data in the data dir (/var/opt/influxdb/data) on my node2. Additionally, my peers.json file, even if I modify it on node2 to have node1 and node2, will always end up with only node2 in it when I restart InfluxDB.

[jwilder]

@derailed Also, your attempt #2 (using peers in the config) is not valid anymore. peers was removed from the config file in 0.9.3 I believe. You always need to join a cluster using -join.

@sandmanahoy Can you post your logs for both node1 and node2?

[beckettsean]

@sandmanahoy do you have any environment variables set in the startup scripts? Specifically INFLUXD_OPTS? Those might be overriding other settings, too.

[beckettsean]

@derailed relying on /etc/hosts definitions has caused problems for others, we don't yet know why. Can you try using a full domain name or IP address?

[derailed]

Tx Jason! Using docker to fire off my instances. You are absolutely
correct. Once I've cleaned up the turds the cluster fired off using the
-join option.

Tx for looking into this!!

On Tue, Oct 13, 2015 at 1:20 PM, Jason Wilder [email protected]
wrote:

@derailed https://github.com/derailed Also, your attempt #2
#2 (using peers in the
config) is not valid anymore. peers was removed from the config file in
0.9.3 I believe. You always need to join a cluster using -join.

@sandmanahoy https://github.com/sandmanahoy Can you post your logs for
both node1 and node2?

—
Reply to this email directly or view it on GitHub
#4429 (comment).

[sandmanahoy]

/etc/default/influxdb on node2 (node1=TSInfluxA, node2=TSInfluxB):

Daemon options

INFLUXD_OPTS="-join TSInfluxA.cdnqa.com:8088"

ps -elf on node2:
4 S root 2279 1 0 80 0 - 87882 futex_ 19:38 pts/0 00:00:00 /opt/influxdb/influxd -pidfile /var/run/influxdb/influxd.pid -config /etc/opt/influxdb/influxdb.conf -join TSInfluxA.cdnqa.com:8088

Logs won't let me attach, but I see this on node2, like OP did:
[metastore] 2015/10/13 20:05:41 Skipping cluster join: already member of cluster: nodeId=1 raftEnabled=true peers=[TSInfluxB.cdnqa.com:8088]

Is there anything else specific from the logs you'd like to see? Thanks again for your help btw :)

[jwilder]

@sandmanahoy This line:

[metastore] 2015/10/13 20:05:41 Skipping cluster join: already member of cluster: nodeId=1 raftEnabled=true peers=[TSInfluxB.cdnqa.com:8088]

indicates node2 has data in it's meta directory and it has already started as a single node. The logs should show what directory it is using for storage. Try stopping node2, removing all the contents of the data dirs and restarting the process with the -join flag.

[sandmanahoy]

Wiped the meta directory completely (rm -rf /var/opt/influxdb/meta/*) and restarted Influx and now it's working. Thanks again!