Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Zookeeper Cluster can't start when istio is used #102

Closed
sylvainOL opened this issue Nov 27, 2019 · 6 comments · Fixed by #113
Closed

Zookeeper Cluster can't start when istio is used #102

sylvainOL opened this issue Nov 27, 2019 · 6 comments · Fixed by #113
Labels
kind/bug Something isn't working

Comments

@sylvainOL
Copy link

Hello,

I've deployed a kubernetes cluster with istio.
When trying to deploy a 3 nodes ZooKeeper Cluster, the second one can't start because of immediate closed connections.

As banzai cloud has made a blog post on Kafka (+ZK?) on Istio (https://banzaicloud.com/blog/kafka-on-istio-performance/), and they propose to use your operator (https://github.com/banzaicloud/kafka-operator), I assumed it's possible but don't see how :-/

  • kubectl version: 1.15.3
  • Istio: v1.4
  • ZooKeeper Operator: 0.2.0 (from banzaicloud repo)

I've tried to disable mTLS but doesn't seem to be the issue.

here's the way I deployed:

helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com/
helm repo update
kubectl create ns zookeeper
kubectl label namespace zookeeper istio-injection=enabled
helm install --name zookeeper-operator --namespace=zookeeper banzaicloud-stable/zookeeper-operator
kubectl apply -f - <<EOF
apiVersion: zookeeper.pravega.io/v1beta1
kind: ZookeeperCluster
metadata:
  name: onap
  namespace: zookeeper
spec:
  replicas: 3
  persistence:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 3G
    storageClassName: ssd
EOF

here's the logs:

  • node-0:
+ source /conf/env.sh
++ DOMAIN=onap-headless.zookeeper.svc.cluster.local
++ QUORUM_PORT=2888
++ LEADER_PORT=3888
++ CLIENT_HOST=onap-client
++ CLIENT_PORT=2181
+ source /usr/local/bin/zookeeperFunctions.sh
++ set -ex
++ hostname -s
+ HOST=onap-0
+ DATA_DIR=/data
+ MYID_FILE=/data/myid
+ LOG4J_CONF=/conf/log4j-quiet.properties
+ DYNCONFIG=/data/zoo.cfg.dynamic
+ [[ onap-0 =~ (.*)-([0-9]+)$ ]]
+ NAME=onap
+ ORD=0
+ MYID=1
+ WRITE_CONFIGURATION=true
+ REGISTER_NODE=true
+ '[' -f /data/myid ']'
++ cat /data/myid
+ EXISTING_ID=1
+ [[ 1 == \1 ]]
+ WRITE_CONFIGURATION=false
+ set +e
+ nslookup onap-headless.zookeeper.svc.cluster.local
nslookup: can't resolve '(null)': Name does not resolve

nslookup: can't resolve 'onap-headless.zookeeper.svc.cluster.local': Name does not resolve
+ [[ 1 -eq 1 ]]
+ set -e
+ WRITE_CONFIGURATION=true
+ REGISTER_NODE=false
+ [[ true == true ]]
+ echo 'Writing myid: 1 to: /data/myid.'
+ echo 1
Writing myid: 1 to: /data/myid.
+ [[ 1 -eq 1 ]]
+ [[ false == false ]]
+ ROLE=participant
+ echo Initial initialization of ordinal 0 pod, creating new config.
Initial initialization of ordinal 0 pod, creating new config.
++ zkConfig
++ echo 'onap-0.onap-headless.zookeeper.svc.cluster.local:2888:3888:participant;2181'
+ ZKCONFIG='onap-0.onap-headless.zookeeper.svc.cluster.local:2888:3888:participant;2181'
+ echo Writing bootstrap configuration with the following config:
Writing bootstrap configuration with the following config:
onap-0.onap-headless.zookeeper.svc.cluster.local:2888:3888:participant;2181
+ echo 'onap-0.onap-headless.zookeeper.svc.cluster.local:2888:3888:participant;2181'
+ echo 1
+ echo 'server.1=onap-0.onap-headless.zookeeper.svc.cluster.local:2888:3888:participant;2181'
Starting zookeeper service
+ echo 'Starting zookeeper service'
+ zkServer.sh start-foreground
ZooKeeper JMX enabled by default
Using config: /conf/zoo.cfg
2019-11-27 15:22:48,324 [myid:] - INFO  [main:QuorumPeerConfig@130] - Reading configuration from: /conf/zoo.cfg
2019-11-27 15:22:48,329 [myid:] - INFO  [main:QuorumPeerConfig@366] - clientPort is not set
2019-11-27 15:22:48,329 [myid:] - INFO  [main:QuorumPeerConfig@380] - secureClientPort is not set
2019-11-27 15:22:48,335 [myid:] - WARN  [main:QuorumPeerConfig@641] - No server failure will be tolerated. You need at least 3 servers.
2019-11-27 15:22:48,339 [myid:1] - INFO  [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2019-11-27 15:22:48,339 [myid:1] - INFO  [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2019-11-27 15:22:48,339 [myid:1] - INFO  [main:DatadirCleanupManager@101] - Purge task is not scheduled.
2019-11-27 15:22:48,340 [myid:1] - INFO  [main:ManagedUtil@46] - Log4j found with jmx enabled.
2019-11-27 15:22:48,351 [myid:1] - INFO  [main:QuorumPeerMain@141] - Starting quorum peer
2019-11-27 15:22:48,361 [myid:1] - INFO  [main:ServerCnxnFactory@135] - Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory
2019-11-27 15:22:48,364 [myid:1] - INFO  [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 2 worker threads, and 64 kB direct buffers.
2019-11-27 15:22:48,369 [myid:1] - INFO  [main:NIOServerCnxnFactory@686] - binding to port /0.0.0.0:2181
2019-11-27 15:22:48,399 [myid:1] - INFO  [main:Log@186] - Logging initialized @527ms
2019-11-27 15:22:48,460 [myid:1] - WARN  [main:ContextHandler@1339] - o.e.j.s.ServletContextHandler@1d057a39{/,null,null} contextPath ends with /*
2019-11-27 15:22:48,460 [myid:1] - WARN  [main:ContextHandler@1350] - Empty contextPath
2019-11-27 15:22:48,481 [myid:1] - INFO  [main:QuorumPeer@1430] - Local sessions disabled
2019-11-27 15:22:48,482 [myid:1] - INFO  [main:QuorumPeer@1441] - Local session upgrading disabled
2019-11-27 15:22:48,482 [myid:1] - INFO  [main:QuorumPeer@1408] - tickTime set to 2000
2019-11-27 15:22:48,482 [myid:1] - INFO  [main:QuorumPeer@1452] - minSessionTimeout set to 4000
2019-11-27 15:22:48,482 [myid:1] - INFO  [main:QuorumPeer@1463] - maxSessionTimeout set to 40000
2019-11-27 15:22:48,482 [myid:1] - INFO  [main:QuorumPeer@1478] - initLimit set to 10
2019-11-27 15:22:48,496 [myid:1] - INFO  [main:ZKDatabase@117] - zookeeper.snapshotSizeFactor = 0.33
2019-11-27 15:22:48,498 [myid:1] - INFO  [main:QuorumPeer@2036] - QuorumPeer communication is not secured!
2019-11-27 15:22:48,498 [myid:1] - INFO  [main:QuorumPeer@2065] - quorum.cnxn.threads.size set to 20
2019-11-27 15:22:48,505 [myid:1] - INFO  [main:FileSnap@83] - Reading snapshot /data/version-2/snapshot.100000111
2019-11-27 15:22:48,532 [myid:1] - INFO  [main:Server@327] - jetty-9.2.18.v20160721
2019-11-27 15:22:48,561 [myid:1] - INFO  [main:ContextHandler@744] - Started o.e.j.s.ServletContextHandler@1d057a39{/,null,AVAILABLE}
2019-11-27 15:22:48,566 [myid:1] - INFO  [main:AbstractConnector@266] - Started ServerConnector@71bbf57e{HTTP/1.1}{0.0.0.0:8080}
2019-11-27 15:22:48,567 [myid:1] - INFO  [main:Server@379] - Started @698ms
2019-11-27 15:22:48,567 [myid:1] - INFO  [main:JettyAdminServer@112] - Started AdminServer on address 0.0.0.0, port 8080 and command URL /commands
2019-11-27 15:22:48,571 [myid:1] - INFO  [QuorumPeerListener:QuorumCnxManager$Listener@854] - My election bind port: onap-0.onap-headless.zookeeper.svc.cluster.local/10.233.67.152:3888
2019-11-27 15:22:48,580 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):QuorumPeer@1136] - LOOKING
2019-11-27 15:22:48,581 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@893] - New election. My id =  1, proposed zxid=0x20000001e
2019-11-27 15:22:48,584 [myid:1] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@687] - Notification: 2 (message format version), 1 (n.leader), 0x20000001e (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x2 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2019-11-27 15:22:48,787 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1,name2=LeaderElection]
2019-11-27 15:22:48,789 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):QuorumPeer@1224] - LEADING
2019-11-27 15:22:48,795 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Leader@67] - TCP NoDelay set to: true
2019-11-27 15:22:48,797 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Leader@87] - zookeeper.leader.maxConcurrentSnapshots = 10
2019-11-27 15:22:48,797 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Leader@89] - zookeeper.leader.maxConcurrentSnapshotTimeout = 5
2019-11-27 15:22:48,812 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:zookeeper.version=3.5.4-beta-7f51e5b68cf2f80176ff944a9ebd2abbc65e7327, built on 05/11/2018 16:27 GMT
2019-11-27 15:22:48,812 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:host.name=onap-0.onap-headless.zookeeper.svc.cluster.local
2019-11-27 15:22:48,812 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:java.version=1.8.0_181
2019-11-27 15:22:48,812 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:java.vendor=Oracle Corporation
2019-11-27 15:22:48,812 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:java.home=/usr/lib/jvm/java-1.8-openjdk/jre
2019-11-27 15:22:48,812 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:java.class.path=/zookeeper-3.5.4-beta/bin/../build/classes:/zookeeper-3.5.4-beta/bin/../build/lib/*.jar:/zookeeper-3.5.4-beta/bin/../lib/slf4j-log4j12-1.7.25.jar:/zookeeper-3.5.4-beta/bin/../lib/slf4j-api-1.7.25.jar:/zookeeper-3.5.4-beta/bin/../lib/netty-3.10.6.Final.jar:/zookeeper-3.5.4-beta/bin/../lib/log4j-1.2.17.jar:/zookeeper-3.5.4-beta/bin/../lib/jline-2.11.jar:/zookeeper-3.5.4-beta/bin/../lib/jetty-util-9.2.18.v20160721.jar:/zookeeper-3.5.4-beta/bin/../lib/jetty-servlet-9.2.18.v20160721.jar:/zookeeper-3.5.4-beta/bin/../lib/jetty-server-9.2.18.v20160721.jar:/zookeeper-3.5.4-beta/bin/../lib/jetty-security-9.2.18.v20160721.jar:/zookeeper-3.5.4-beta/bin/../lib/jetty-io-9.2.18.v20160721.jar:/zookeeper-3.5.4-beta/bin/../lib/jetty-http-9.2.18.v20160721.jar:/zookeeper-3.5.4-beta/bin/../lib/javax.servlet-api-3.1.0.jar:/zookeeper-3.5.4-beta/bin/../lib/jackson-mapper-asl-1.9.11.jar:/zookeeper-3.5.4-beta/bin/../lib/jackson-core-asl-1.9.11.jar:/zookeeper-3.5.4-beta/bin/../lib/commons-cli-1.2.jar:/zookeeper-3.5.4-beta/bin/../lib/audience-annotations-0.5.0.jar:/zookeeper-3.5.4-beta/bin/../zookeeper-3.5.4-beta.jar:/zookeeper-3.5.4-beta/bin/../src/java/lib/*.jar:/conf:
2019-11-27 15:22:48,813 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:java.library.path=/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64:/usr/lib/jvm/java-1.8-openjdk/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-11-27 15:22:48,813 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:java.io.tmpdir=/tmp
2019-11-27 15:22:48,813 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:java.compiler=<NA>
2019-11-27 15:22:48,813 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:os.name=Linux
2019-11-27 15:22:48,813 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:os.arch=amd64
2019-11-27 15:22:48,813 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:os.version=4.9.0-8-amd64
2019-11-27 15:22:48,814 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:user.name=root
2019-11-27 15:22:48,814 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:user.home=/root
2019-11-27 15:22:48,814 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:user.dir=/zookeeper-3.5.4-beta
2019-11-27 15:22:48,814 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:os.memory.free=444MB
2019-11-27 15:22:48,814 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:os.memory.max=966MB
2019-11-27 15:22:48,814 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:os.memory.total=487MB
2019-11-27 15:22:48,816 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ZooKeeperServer@928] - minSessionTimeout set to 4000
2019-11-27 15:22:48,816 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ZooKeeperServer@937] - maxSessionTimeout set to 40000
2019-11-27 15:22:48,817 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ZooKeeperServer@160] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /data/version-2 snapdir /data/version-2
2019-11-27 15:22:48,818 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Leader@432] - LEADING - LEADER ELECTION TOOK - 30 MS
2019-11-27 15:22:48,820 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FileTxnSnapLog@367] - Snapshotting: 0x20000001e to /data/version-2/snapshot.20000001e
2019-11-27 15:22:48,831 [myid:1] - ERROR [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):QuorumPeer@1564] - Error writing next dynamic config file to disk:
2019-11-27 15:22:48,836 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Leader@1264] - Have quorum of supporters, sids: [ [1],[1] ]; starting up and setting last processed zxid: 0x300000000
2019-11-27 15:22:48,836 [myid:1] - ERROR [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):QuorumPeer@1596] - Error closing file:
2019-11-27 15:22:48,847 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):CommitProcessor@255] - Configuring CommitProcessor with 1 worker threads.
2019-11-27 15:22:48,856 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):PrepRequestProcessor@99] - zookeeper.skipACL=="yes", ACL checks will be skipped
2019-11-27 15:22:48,858 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ContainerManager@64] - Using checkIntervalMs=60000 maxPerMinute=10000
2019-11-27 15:22:59,878 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:40039
2019-11-27 15:22:59,893 [myid:1] - INFO  [NIOWorkerThread-1:FourLetterCommands@234] - The list of known four letter word commands is : [{1936881266=srvr, 1937006964=stat, 2003003491=wchc, 1685417328=dump, 1668445044=crst, 1936880500=srst, 1701738089=envi, 1668247142=conf, -720899=telnet close, 2003003507=wchs, 2003003504=wchp, 1684632179=dirs, 1668247155=cons, 1835955314=mntr, 1769173615=isro, 1920298859=ruok, 1735683435=gtmk, 1937010027=stmk}]
2019-11-27 15:22:59,893 [myid:1] - INFO  [NIOWorkerThread-1:FourLetterCommands@235] - The list of enabled four letter word commands is : [[stat, envi, conf, ruok, mntr, srvr, crst, cons]]
2019-11-27 15:22:59,893 [myid:1] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:40039
2019-11-27 15:22:59,899 [myid:1] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:40039 (no session established for client)
2019-11-27 15:23:00,535 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:44549
2019-11-27 15:23:00,537 [myid:1] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:44549
2019-11-27 15:23:00,539 [myid:1] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:44549 (no session established for client)
2019-11-27 15:23:09,895 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:42815
2019-11-27 15:23:09,897 [myid:1] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:42815
2019-11-27 15:23:09,898 [myid:1] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:42815 (no session established for client)
2019-11-27 15:23:10,549 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:35831
2019-11-27 15:23:10,551 [myid:1] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:35831
2019-11-27 15:23:10,551 [myid:1] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:35831 (no session established for client)
2019-11-27 15:23:19,893 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:45287
2019-11-27 15:23:19,896 [myid:1] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:45287
2019-11-27 15:23:19,897 [myid:1] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:45287 (no session established for client)
2019-11-27 15:23:20,517 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:43115
2019-11-27 15:23:20,519 [myid:1] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:43115
2019-11-27 15:23:20,520 [myid:1] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:43115 (no session established for client)
2019-11-27 15:23:23,856 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:33672
2019-11-27 15:23:23,862 [myid:1] - INFO  [NIOWorkerThread-1:ZooKeeperServer@1041] - Client attempting to establish new session at /127.0.0.1:33672
2019-11-27 15:23:23,864 [myid:1] - INFO  [SyncThread:1:FileTxnLog@201] - Creating new log file: log.300000001
2019-11-27 15:23:23,877 [myid:1] - INFO  [CommitProcWorkThread-1:ZooKeeperServer@748] - Established session 0x1000bc750090000 with negotiated timeout 4000 for client /127.0.0.1:33672
2019-11-27 15:23:24,224 [myid:1] - WARN  [NIOWorkerThread-1:NIOServerCnxn@366] - Unable to read additional data from client sessionid 0x1000bc750090000, likely client has closed socket
2019-11-27 15:23:24,225 [myid:1] - INFO  [NIOWorkerThread-1:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1,name2=Leader,name3=Connections,name4=127.0.0.1,name5=0x1000bc750090000]
2019-11-27 15:23:24,226 [myid:1] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:33672 which had sessionid 0x1000bc750090000
2019-11-27 15:23:24,613 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:33678
2019-11-27 15:23:24,615 [myid:1] - INFO  [NIOWorkerThread-2:ZooKeeperServer@1041] - Client attempting to establish new session at /127.0.0.1:33678
2019-11-27 15:23:24,620 [myid:1] - INFO  [CommitProcWorkThread-1:ZooKeeperServer@748] - Established session 0x1000bc750090001 with negotiated timeout 4000 for client /127.0.0.1:33678
2019-11-27 15:23:24,635 [myid:1] - WARN  [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@432] - skipACL is set, reconfig operation will skip ACL checks!
2019-11-27 15:23:24,635 [myid:1] - INFO  [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@474] - Incremental reconfig
2019-11-27 15:23:24,639 [myid:1] - WARN  [ProcessThread(sid:1 cport:-1)::QuorumPeer$QuorumServer@173] - Failed to resolve address: onap-1.onap-headless.zookeeper.svc.cluster.local
java.net.UnknownHostException: onap-1.onap-headless.zookeeper.svc.cluster.local
	at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
	at java.net.InetAddress.getAllByName(InetAddress.java:1192)
	at java.net.InetAddress.getAllByName(InetAddress.java:1126)
	at java.net.InetAddress.getByName(InetAddress.java:1076)
	at org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer.recreateSocketAddresses(QuorumPeer.java:171)
	at org.apache.zookeeper.server.quorum.QuorumPeer.recreateSocketAddresses(QuorumPeer.java:727)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:682)
	at org.apache.zookeeper.server.quorum.QuorumPeer.connectNewPeers(QuorumPeer.java:1519)
	at org.apache.zookeeper.server.quorum.QuorumPeer.setLastSeenQuorumVerifier(QuorumPeer.java:1556)
	at org.apache.zookeeper.server.quorum.Leader.propose(Leader.java:1074)
	at org.apache.zookeeper.server.quorum.ProposalRequestProcessor.processRequest(ProposalRequestProcessor.java:78)
	at org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:906)
	at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:145)
2019-11-27 15:23:24,644 [myid:1] - WARN  [ProcessThread(sid:1 cport:-1)::QuorumCnxManager@660] - Cannot open channel to 2 at election address onap-1.onap-headless.zookeeper.svc.cluster.local:3888
java.net.UnknownHostException: onap-1.onap-headless.zookeeper.svc.cluster.local
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:638)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:695)
	at org.apache.zookeeper.server.quorum.QuorumPeer.connectNewPeers(QuorumPeer.java:1519)
	at org.apache.zookeeper.server.quorum.QuorumPeer.setLastSeenQuorumVerifier(QuorumPeer.java:1556)
	at org.apache.zookeeper.server.quorum.Leader.propose(Leader.java:1074)
	at org.apache.zookeeper.server.quorum.ProposalRequestProcessor.processRequest(ProposalRequestProcessor.java:78)
	at org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:906)
	at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:145)
2019-11-27 15:23:24,645 [myid:1] - ERROR [ProcessThread(sid:1 cport:-1)::QuorumPeer@1564] - Error writing next dynamic config file to disk:
2019-11-27 15:23:24,647 [myid:1] - ERROR [SyncThread:1:QuorumPeer@1596] - Error closing file:
2019-11-27 15:23:24,649 [myid:1] - WARN  [SyncThread:1:QuorumPeer@1527] - Restarting Leader Election
2019-11-27 15:23:24,650 [myid:1] - INFO  [onap-0.onap-headless.zookeeper.svc.cluster.local/10.233.67.152:3888:QuorumCnxManager$Listener@892] - Leaving listener
2019-11-27 15:23:24,651 [myid:1] - INFO  [QuorumPeerListener:QuorumCnxManager$Listener@854] - My election bind port: onap-0.onap-headless.zookeeper.svc.cluster.local/10.233.67.152:3888
2019-11-27 15:23:25,005 [myid:1] - WARN  [NIOWorkerThread-2:NIOServerCnxn@366] - Unable to read additional data from client sessionid 0x1000bc750090001, likely client has closed socket
2019-11-27 15:23:25,006 [myid:1] - INFO  [NIOWorkerThread-2:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1,name2=Leader,name3=Connections,name4=127.0.0.1,name5=0x1000bc750090001]
2019-11-27 15:23:25,007 [myid:1] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:33678 which had sessionid 0x1000bc750090001
2019-11-27 15:23:27,586 [myid:1] - INFO  [WorkerSender[myid=1]:FastLeaderElection$Messenger$WorkerSender@469] - WorkerSender is down
2019-11-27 15:23:27,588 [myid:1] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@439] - WorkerReceiver is down
2019-11-27 15:23:29,311 [myid:1] - INFO  [SessionTracker:ZooKeeperServer@392] - Expiring session 0x1000bc750090000, timeout of 4000ms exceeded
2019-11-27 15:23:29,311 [myid:1] - INFO  [SessionTracker:QuorumZooKeeperServer@157] - Submitting global closeSession request for session 0x1000bc750090000
2019-11-27 15:23:29,311 [myid:1] - INFO  [SessionTracker:ZooKeeperServer@392] - Expiring session 0x1000bc750090001, timeout of 4000ms exceeded
2019-11-27 15:23:29,312 [myid:1] - INFO  [SessionTracker:QuorumZooKeeperServer@157] - Submitting global closeSession request for session 0x1000bc750090001
2019-11-27 15:23:29,312 [myid:1] - INFO  [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@611] - Processed session termination for sessionid: 0x1000bc750090000
2019-11-27 15:23:29,312 [myid:1] - INFO  [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@611] - Processed session termination for sessionid: 0x1000bc750090001
2019-11-27 15:23:29,885 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:45507
2019-11-27 15:23:29,887 [myid:1] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:45507
2019-11-27 15:23:29,888 [myid:1] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:45507 (no session established for client)
2019-11-27 15:23:30,528 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:35905
2019-11-27 15:23:30,529 [myid:1] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:35905
2019-11-27 15:23:30,530 [myid:1] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:35905 (no session established for client)
2019-11-27 15:23:39,773 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:33802
2019-11-27 15:23:39,775 [myid:1] - INFO  [NIOWorkerThread-1:ZooKeeperServer@1041] - Client attempting to establish new session at /127.0.0.1:33802
2019-11-27 15:23:39,780 [myid:1] - INFO  [CommitProcWorkThread-1:ZooKeeperServer@748] - Established session 0x1000bc750090002 with negotiated timeout 4000 for client /127.0.0.1:33802
2019-11-27 15:23:39,882 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:39623
...
  • node-1
+ source /conf/env.sh
++ DOMAIN=onap-headless.zookeeper.svc.cluster.local
++ QUORUM_PORT=2888
++ LEADER_PORT=3888
++ CLIENT_HOST=onap-client
++ CLIENT_PORT=2181
+ source /usr/local/bin/zookeeperFunctions.sh
++ set -ex
++ hostname -s
+ HOST=onap-1
+ DATA_DIR=/data
+ MYID_FILE=/data/myid
+ LOG4J_CONF=/conf/log4j-quiet.properties
+ DYNCONFIG=/data/zoo.cfg.dynamic
+ [[ onap-1 =~ (.*)-([0-9]+)$ ]]
+ NAME=onap
+ ORD=1
+ MYID=2
+ WRITE_CONFIGURATION=true
+ REGISTER_NODE=true
+ '[' -f /data/myid ']'
++ cat /data/myid
+ EXISTING_ID=2
+ [[ 2 == \2 ]]
+ WRITE_CONFIGURATION=false
+ set +e
+ nslookup onap-headless.zookeeper.svc.cluster.local
nslookup: can't resolve '(null)': Name does not resolve

Name:      onap-headless.zookeeper.svc.cluster.local
Address 1: 10.233.67.152 10-233-67-152.onap-client.zookeeper.svc.cluster.local
+ [[ 0 -eq 1 ]]
+ set -e
+ set +e
++ zkConnectionString
++ set +e
++ nslookup onap-client
++ [[ 0 -eq 1 ]]
++ set -e
++ echo onap-client:2181
+ ZKURL=onap-client:2181
+ set -e
++ java -Dlog4j.configuration=file:/conf/log4j-quiet.properties -jar /root/zu.jar get-all onap-client:2181
Connecting to Zookeeper onap-client:2181
+ CONFIG='server.1=onap-0.onap-headless.zookeeper.svc.cluster.local:2888:3888:participant;0.0.0.0:2181
version=300000000'
++ echo 'server.1=onap-0.onap-headless.zookeeper.svc.cluster.local:2888:3888:participant;0.0.0.0:2181
version=300000000'
++ grep server.2=
++ wc -l
+ REGISTERED=0
+ [[ 0 -eq 1 ]]
+ REGISTER_NODE=true
+ WRITE_CONFIGURATION=true
Writing myid: 2 to: /data/myid.
+ [[ true == true ]]
+ echo 'Writing myid: 2 to: /data/myid.'
+ echo 2
+ [[ 2 -eq 1 ]]
+ [[ 2 -ne 1 ]]
+ [[ true == false ]]
+ [[ true == true ]]
+ ROLE=observer
++ zkConfig
++ echo 'onap-1.onap-headless.zookeeper.svc.cluster.local:2888:3888:observer;2181'
+ ZKCONFIG='onap-1.onap-headless.zookeeper.svc.cluster.local:2888:3888:observer;2181'
+ echo Registering node and writing local configuration to disk.
+ java -Dlog4j.configuration=file:/conf/log4j-quiet.properties -jar /root/zu.jar add onap-client:2181 2 'onap-1.onap-headless.zookeeper.svc.cluster.local:2888:3888:observer;2181' /data/zoo.cfg.dynamic
Registering node and writing local configuration to disk.
Connecting to Zookeeper onap-client:2181
+ echo 'Starting zookeeper service'
+ zkServer.sh start-foreground
Starting zookeeper service
ZooKeeper JMX enabled by default
Using config: /conf/zoo.cfg
2019-11-27 15:23:25,496 [myid:] - INFO  [main:QuorumPeerConfig@130] - Reading configuration from: /conf/zoo.cfg
2019-11-27 15:23:25,502 [myid:] - INFO  [main:QuorumPeerConfig@366] - clientPort is not set
2019-11-27 15:23:25,502 [myid:] - INFO  [main:QuorumPeerConfig@380] - secureClientPort is not set
2019-11-27 15:23:25,510 [myid:] - WARN  [main:QuorumPeerConfig@641] - No server failure will be tolerated. You need at least 3 servers.
2019-11-27 15:23:25,513 [myid:2] - WARN  [main:QuorumPeerConfig@706] - Peer type from servers list (OBSERVER) doesn't match peerType (PARTICIPANT). Defaulting to servers list.
2019-11-27 15:23:25,515 [myid:2] - INFO  [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2019-11-27 15:23:25,515 [myid:2] - INFO  [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2019-11-27 15:23:25,515 [myid:2] - INFO  [main:DatadirCleanupManager@101] - Purge task is not scheduled.
2019-11-27 15:23:25,516 [myid:2] - INFO  [main:ManagedUtil@46] - Log4j found with jmx enabled.
2019-11-27 15:23:25,528 [myid:2] - INFO  [main:QuorumPeerMain@141] - Starting quorum peer
2019-11-27 15:23:25,540 [myid:2] - INFO  [main:ServerCnxnFactory@135] - Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory
2019-11-27 15:23:25,545 [myid:2] - INFO  [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 2 worker threads, and 64 kB direct buffers.
2019-11-27 15:23:25,553 [myid:2] - INFO  [main:NIOServerCnxnFactory@686] - binding to port /0.0.0.0:2181
2019-11-27 15:23:25,584 [myid:2] - INFO  [main:Log@186] - Logging initialized @552ms
2019-11-27 15:23:25,641 [myid:2] - WARN  [main:ContextHandler@1339] - o.e.j.s.ServletContextHandler@1d057a39{/,null,null} contextPath ends with /*
2019-11-27 15:23:25,641 [myid:2] - WARN  [main:ContextHandler@1350] - Empty contextPath
2019-11-27 15:23:25,659 [myid:2] - INFO  [main:QuorumPeer@1430] - Local sessions disabled
2019-11-27 15:23:25,660 [myid:2] - INFO  [main:QuorumPeer@1441] - Local session upgrading disabled
2019-11-27 15:23:25,660 [myid:2] - INFO  [main:QuorumPeer@1408] - tickTime set to 2000
2019-11-27 15:23:25,660 [myid:2] - INFO  [main:QuorumPeer@1452] - minSessionTimeout set to 4000
2019-11-27 15:23:25,660 [myid:2] - INFO  [main:QuorumPeer@1463] - maxSessionTimeout set to 40000
2019-11-27 15:23:25,660 [myid:2] - INFO  [main:QuorumPeer@1478] - initLimit set to 10
2019-11-27 15:23:25,673 [myid:2] - INFO  [main:ZKDatabase@117] - zookeeper.snapshotSizeFactor = 0.33
2019-11-27 15:23:25,675 [myid:2] - INFO  [main:QuorumPeer@2036] - QuorumPeer communication is not secured!
2019-11-27 15:23:25,675 [myid:2] - INFO  [main:QuorumPeer@2065] - quorum.cnxn.threads.size set to 20
2019-11-27 15:23:25,679 [myid:2] - INFO  [main:FileSnap@83] - Reading snapshot /data/version-2/snapshot.0
2019-11-27 15:23:25,693 [myid:2] - INFO  [main:Server@327] - jetty-9.2.18.v20160721
2019-11-27 15:23:25,730 [myid:2] - INFO  [main:ContextHandler@744] - Started o.e.j.s.ServletContextHandler@1d057a39{/,null,AVAILABLE}
2019-11-27 15:23:25,735 [myid:2] - INFO  [main:AbstractConnector@266] - Started ServerConnector@6737fd8f{HTTP/1.1}{0.0.0.0:8080}
2019-11-27 15:23:25,736 [myid:2] - INFO  [main:Server@379] - Started @707ms
2019-11-27 15:23:25,736 [myid:2] - INFO  [main:JettyAdminServer@112] - Started AdminServer on address 0.0.0.0, port 8080 and command URL /commands
2019-11-27 15:23:25,742 [myid:2] - INFO  [QuorumPeerListener:QuorumCnxManager$Listener@854] - My election bind port: onap-1.onap-headless.zookeeper.svc.cluster.local/10.233.66.89:3888
2019-11-27 15:23:25,752 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):QuorumPeer@1136] - LOOKING
2019-11-27 15:23:25,753 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@893] - New election. My id =  2, proposed zxid=0x8000000000000000
2019-11-27 15:23:25,760 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1148] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
	at java.io.DataInputStream.readInt(DataInputStream.java:392)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1133)
2019-11-27 15:23:25,762 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1151] - Interrupting SendWorker
2019-11-27 15:23:25,762 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1064] - Interrupted while waiting for message on queue
java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1215)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:75)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1052)
2019-11-27 15:23:25,763 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1074] - Send worker leaving thread  id 1 my id = 2
2019-11-27 15:23:25,962 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@927] - Notification time out: 400
2019-11-27 15:23:25,965 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1148] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
	at java.io.DataInputStream.readInt(DataInputStream.java:392)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1133)
2019-11-27 15:23:25,965 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1151] - Interrupting SendWorker
2019-11-27 15:23:25,966 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1064] - Interrupted while waiting for message on queue
java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1215)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:75)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1052)
2019-11-27 15:23:25,967 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1074] - Send worker leaving thread  id 1 my id = 2
2019-11-27 15:23:26,363 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@927] - Notification time out: 800
2019-11-27 15:23:26,367 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1148] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
	at java.io.DataInputStream.readInt(DataInputStream.java:392)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1133)
2019-11-27 15:23:26,368 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1151] - Interrupting SendWorker
2019-11-27 15:23:26,368 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1064] - Interrupted while waiting for message on queue
java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1215)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:75)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1052)
2019-11-27 15:23:26,369 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1074] - Send worker leaving thread  id 1 my id = 2
2019-11-27 15:23:27,164 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@927] - Notification time out: 1600
2019-11-27 15:23:27,168 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1148] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
	at java.io.DataInputStream.readInt(DataInputStream.java:392)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1133)
2019-11-27 15:23:27,168 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1151] - Interrupting SendWorker
2019-11-27 15:23:27,169 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1064] - Interrupted while waiting for message on queue
java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1215)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:75)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1052)
2019-11-27 15:23:27,169 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1074] - Send worker leaving thread  id 1 my id = 2
2019-11-27 15:23:28,766 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@927] - Notification time out: 3200
2019-11-27 15:23:28,769 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1148] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
	at java.io.DataInputStream.readInt(DataInputStream.java:392)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1133)
2019-11-27 15:23:28,770 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1151] - Interrupting SendWorker
2019-11-27 15:23:28,770 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1064] - Interrupted while waiting for message on queue
java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1215)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:75)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1052)
2019-11-27 15:23:28,771 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1074] - Send worker leaving thread  id 1 my id = 2
2019-11-27 15:23:31,967 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@927] - Notification time out: 6400
2019-11-27 15:23:31,970 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1148] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
	at java.io.DataInputStream.readInt(DataInputStream.java:392)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1133)
2019-11-27 15:23:31,971 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1151] - Interrupting SendWorker
2019-11-27 15:23:31,971 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1064] - Interrupted while waiting for message on queue
java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1215)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:75)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1052)
2019-11-27 15:23:31,972 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1074] - Send worker leaving thread  id 1 my id = 2
2019-11-27 15:23:37,591 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:36965
2019-11-27 15:23:37,603 [myid:2] - INFO  [NIOWorkerThread-1:FourLetterCommands@234] - The list of known four letter word commands is : [{1936881266=srvr, 1937006964=stat, 2003003491=wchc, 1685417328=dump, 1668445044=crst, 1936880500=srst, 1701738089=envi, 1668247142=conf, -720899=telnet close, 2003003507=wchs, 2003003504=wchp, 1684632179=dirs, 1668247155=cons, 1835955314=mntr, 1769173615=isro, 1920298859=ruok, 1735683435=gtmk, 1937010027=stmk}]
2019-11-27 15:23:37,603 [myid:2] - INFO  [NIOWorkerThread-1:FourLetterCommands@235] - The list of enabled four letter word commands is : [[stat, envi, conf, ruok, mntr, srvr, crst, cons]]
2019-11-27 15:23:37,604 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:36965
2019-11-27 15:23:37,610 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:36965 (no session established for client)
2019-11-27 15:23:38,367 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@927] - Notification time out: 12800
2019-11-27 15:23:38,374 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1148] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
	at java.io.DataInputStream.readInt(DataInputStream.java:392)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1133)
2019-11-27 15:23:38,374 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1151] - Interrupting SendWorker
2019-11-27 15:23:38,374 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1069] - Exception when using channel: for id 1 my id = 2 error = java.net.SocketException: Broken pipe (Write failed)
2019-11-27 15:23:38,375 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1074] - Send worker leaving thread  id 1 my id = 2
2019-11-27 15:23:39,383 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:46681
2019-11-27 15:23:39,384 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:46681
2019-11-27 15:23:39,385 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:46681 (no session established for client)
2019-11-27 15:23:47,574 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:44881
2019-11-27 15:23:47,575 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:44881
2019-11-27 15:23:47,576 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:44881 (no session established for client)
2019-11-27 15:23:49,386 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:44397
2019-11-27 15:23:49,387 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:44397
2019-11-27 15:23:49,388 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:44397 (no session established for client)
2019-11-27 15:23:51,168 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@927] - Notification time out: 25600
2019-11-27 15:23:51,174 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1148] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
	at java.io.DataInputStream.readInt(DataInputStream.java:392)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1133)
2019-11-27 15:23:51,174 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1151] - Interrupting SendWorker
2019-11-27 15:23:51,174 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1069] - Exception when using channel: for id 1 my id = 2 error = java.net.SocketException: Broken pipe (Write failed)
2019-11-27 15:23:51,175 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1074] - Send worker leaving thread  id 1 my id = 2
2019-11-27 15:23:57,581 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:40065
2019-11-27 15:23:57,583 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:40065
2019-11-27 15:23:57,584 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:40065 (no session established for client)
2019-11-27 15:23:59,400 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:36819
2019-11-27 15:23:59,401 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:36819
2019-11-27 15:23:59,402 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:36819 (no session established for client)
2019-11-27 15:24:07,572 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:40383
2019-11-27 15:24:07,575 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:40383
2019-11-27 15:24:07,575 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:40383 (no session established for client)
2019-11-27 15:24:09,397 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:45489
2019-11-27 15:24:09,399 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:45489
2019-11-27 15:24:09,399 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:45489 (no session established for client)
2019-11-27 15:24:16,769 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@927] - Notification time out: 51200
2019-11-27 15:24:16,778 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1148] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
	at java.io.DataInputStream.readInt(DataInputStream.java:392)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1133)
2019-11-27 15:24:16,778 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1151] - Interrupting SendWorker
2019-11-27 15:24:16,778 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1069] - Exception when using channel: for id 1 my id = 2 error = java.net.SocketException: Broken pipe (Write failed)
2019-11-27 15:24:16,780 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1074] - Send worker leaving thread  id 1 my id = 2
2019-11-27 15:24:17,577 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:38589
2019-11-27 15:24:17,579 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:38589
2019-11-27 15:24:17,579 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:38589 (no session established for client)
2019-11-27 15:24:19,383 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:37545
2019-11-27 15:24:19,385 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:37545
2019-11-27 15:24:19,385 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:37545 (no session established for client)
2019-11-27 15:24:27,578 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:43155
2019-11-27 15:24:27,579 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:43155
2019-11-27 15:24:27,579 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:43155 (no session established for client)
2019-11-27 15:24:29,392 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:44137
2019-11-27 15:24:29,393 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:44137
2019-11-27 15:24:29,393 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:44137 (no session established for client)
2019-11-27 15:24:37,575 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:38211
2019-11-27 15:24:37,576 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:38211
2019-11-27 15:24:37,576 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:38211 (no session established for client)
2019-11-27 15:24:39,395 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:43741
2019-11-27 15:24:39,396 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:43741
2019-11-27 15:24:39,396 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:43741 (no session established for client)
2019-11-27 15:24:47,586 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:35941
2019-11-27 15:24:47,587 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:35941
2019-11-27 15:24:47,587 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:35941 (no session established for client)
2019-11-27 15:24:49,391 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:43899
2019-11-27 15:24:49,392 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:43899
2019-11-27 15:24:49,392 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:43899 (no session established for client)
2019-11-27 15:24:57,572 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:43943
2019-11-27 15:24:57,574 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:43943
2019-11-27 15:24:57,574 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:43943 (no session established for client)
2019-11-27 15:24:59,395 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:41531
2019-11-27 15:24:59,395 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:41531
2019-11-27 15:24:59,396 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:41531 (no session established for client)
2019-11-27 15:25:07,592 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:42705
2019-11-27 15:25:07,593 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:42705
2019-11-27 15:25:07,593 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:42705 (no session established for client)
2019-11-27 15:25:07,970 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@927] - Notification time out: 60000
2019-11-27 15:25:07,975 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1148] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
	at java.io.DataInputStream.readInt(DataInputStream.java:392)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1133)
2019-11-27 15:25:07,975 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1151] - Interrupting SendWorker
2019-11-27 15:25:07,976 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1064] - Interrupted while waiting for message on queue
java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1215)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:75)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1052)
2019-11-27 15:25:07,977 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1074] - Send worker leaving thread  id 1 my id = 2
2019-11-27 15:25:09,384 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:38363
2019-11-27 15:25:09,385 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:38363
2019-11-27 15:25:09,385 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:38363 (no session established for client)
...
@sylvainOL sylvainOL mentioned this issue Nov 27, 2019
Closed
@seecsea
Copy link

seecsea commented Dec 5, 2019 •
edited
Loading

Occur the same:
I use the yaml from git repo,not helm
k8s:1.14.5

Readiness probe failed: HTTP probe failed with statuscode: 503

kubectl describe pod zookeeper-operator-8449df9744-lkx9j
Events:
Type Reason Age From Message

Normal Scheduled 2d11h default-scheduler Successfully assigned default/zookeeper-operator-8449df9744-lkx9j to k8stian-n3
Normal Pulled 2d11h kubelet, k8stian-n3 Container image "docker.io/istio/proxy_init:1.2.2" already present on machine
Normal Created 2d11h kubelet, k8stian-n3 Created container istio-init
Normal Started 2d11h kubelet, k8stian-n3 Started container istio-init
Normal Pulled 2d11h kubelet, k8stian-n3 Container image "docker.io/istio/proxyv2:1.2.2" already present on machine
Normal Started 2d11h kubelet, k8stian-n3 Started container istio-proxy
Normal Created 2d11h kubelet, k8stian-n3 Created container istio-proxy
Normal Pulling 2d11h (x2 over 2d11h) kubelet, k8stian-n3 Pulling image "pravega/zookeeper-operator:latest"
Normal Pulled 2d11h (x2 over 2d11h) kubelet, k8stian-n3 Successfully pulled image "pravega/zookeeper-operator:latest"
Normal Created 2d11h (x2 over 2d11h) kubelet, k8stian-n3 Created container zookeeper-operator
Normal Started 2d11h (x2 over 2d11h) kubelet, k8stian-n3 Started container zookeeper-operator
Warning Unhealthy 2m18s (x106771 over 2d11h) kubelet, k8stian-n3 Readiness probe failed: HTTP probe failed with statuscode: 503

[root@k8stian-m2:/usr/local/src/deploy/zookeeper-operator/deploy/crds]# kubectl logs -f zookeeper-operator-8449df9744-lkx9j
Error from server (BadRequest): a container name must be specified for pod zookeeper-operator-8449df9744-lkx9j, choose one of: [zookeeper-operator istio-proxy] or one of the init containers: [istio-init]
[root@k8stian-m2:/usr/local/src/deploy/zookeeper-operator/deploy/crds]# kubectl logs -f zookeeper-operator-8449df9744-lkx9j -c zookeeper-operator
{"level":"info","ts":1575336376.1156788,"logger":"cmd","msg":"zookeeper-operator Version: 0.2.3-17"}
{"level":"info","ts":1575336376.1157143,"logger":"cmd","msg":"Git SHA: cea93ee"}
{"level":"info","ts":1575336376.1157634,"logger":"cmd","msg":"Go Version: go1.12.10"}
{"level":"info","ts":1575336376.1157687,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1575336376.1157732,"logger":"cmd","msg":"operator-sdk Version: v0.3.0"}
{"level":"info","ts":1575336376.1163554,"logger":"leader","msg":"Trying to become the leader."}
{"level":"info","ts":1575336376.294069,"logger":"leader","msg":"No pre-existing lock was found."}
{"level":"info","ts":1575336376.3139248,"logger":"leader","msg":"Became the leader."}
{"level":"info","ts":1575336376.3809543,"logger":"cmd","msg":"Registering Components."}
{"level":"info","ts":1575336376.3813946,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"zookeepercluster-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1575336376.3815124,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"zookeepercluster-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1575336376.3815713,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"zookeepercluster-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1575336376.381645,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"zookeepercluster-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1575336376.3816953,"logger":"cmd","msg":"Starting the Cmd."}
{"level":"info","ts":1575336376.5820937,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"zookeepercluster-controller"}
{"level":"info","ts":1575336376.6822746,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"zookeepercluster-controller","worker count":1}

@spiegela spiegela added the kind/bug Something isn't working label Dec 5, 2019
@spiegela
Copy link
Contributor

spiegela commented Dec 5, 2019

@seecsea @sylvainOL Thanks for reporting this. I'll take a look ASAP.

@sylvainOL sylvainOL mentioned this issue Dec 17, 2019
Closed
@baluchicken baluchicken mentioned this issue Jan 1, 2020
Merged
@alihalabyah
Copy link

Any updates here?

@pbelgundi
Copy link
Contributor

pbelgundi commented May 22, 2020 •
edited
Loading

To fix this we perhaps need to set quorumListenOnAllIPs to true in Zookeeper:

https://github.com/istio/istio.io/blob/master/content/en/faq/applications/zookeeper.md

helm/charts#17183

istio/istio#19280

@AceHack
Copy link

AceHack commented May 24, 2020

Yes it's still not working for me as well.

@AceHack
Copy link

AceHack commented May 24, 2020

When you turn on quorumListenOnAllIPs you are going to start having this problem but if you use my health checks it will work around the problem.

#128

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Something isn't working
Projects
None yet
Milestone
No milestone
6 participants
@spiegela @AceHack @alihalabyah @seecsea @sylvainOL @pbelgundi