Nomad Client: fatal error: concurrent map read and map write #4154

ivanprostran · 2018-04-15T13:24:37Z

Nomad version

Output from nomad version:

Nomad v0.8.0 (751b42a)

Client: true
         Log Level: INFO
            Region: ipdr (DC: dc1)
            Server: false
           Version: 0.8.0

client.hcl

data_dir = "/opt/nomad/data/client/"
client {
  enabled       = true
  cpu_total_compute = 20000
  reserved {
                cpu = 5000
                memory = 28500
                disk = 1
        }
  network_speed = 10000
  servers = ["127.0.0.1:4647"]
  options {
    "driver.raw_exec.enable" = "1"
  }
}
ports {
    http = 5656
}

Output from 'consul version':

Consul v1.0.6
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when > speaking to compatible agents)

Operating system and Environment details

Red Hat Enterprise Linux Server release 7.2 (Maipo)
Linux node03 3.10.0-327.10.1.el7.x86_64 #1 SMP Sat Jan 23 04:54:55 EST 2016 x86_64 x86_64 x86_64 GNU/Linux

Issue

The Nomad Client exits after some time with the following error:
fatal error: concurrent map read and map write

Nomad cluster setup is as follows:

10 nodes (Every node has: nomad client, nomad server, consul)
Client driver being used is "raw_exec " ( i.e. no isolation, no docker, no cgroup)

Reproduction steps

Everything works smoothly and, unfortunately, after some time server stops working.

Notice: This was not the case with the previous version:

Nomad v0.7.1 (0b295d3)

Nomad Server logs

Nothing in server log.

Nomad Client logs

fatal error: concurrent map read and map write

goroutine 99 [running]:
runtime.throw(0x16f1387, 0x21)
	/usr/local/go/src/runtime/panic.go:605 +0x95 fp=0xc4204c7b70 sp=0xc4204c7b50 pc=0x42dca5
runtime.mapaccess2_faststr(0x14a27a0, 0xc4204ed3b0, 0x16e3937, 0x18, 0x4131b8, 0x10)
	/usr/local/go/src/runtime/hashmap_fast.go:324 +0x47a fp=0xc4204c7bc8 sp=0xc4204c7b70 pc=0x40e43a
github.com/hashicorp/nomad/client/driver.cgroupsMounted(...)
	/opt/gopath/src/github.com/hashicorp/nomad/client/driver/utils.go:29
github.com/hashicorp/nomad/client/driver.(*ExecDriver).Fingerprint(0xc4201d4960, 0xc4201d8aa0, 0xc420d27220, 0x4, 0xc4204628b0)
	/opt/gopath/src/github.com/hashicorp/nomad/client/driver/exec_linux.go:20 +0x61 fp=0xc4204c7c20 sp=0xc4204c7bc8 pc=0xe0f3d1
github.com/hashicorp/nomad/client.(*FingerprintManager).fingerprintDriver(0xc4201d2940, 0x16c5a59, 0x4, 0x7f6af40c21a8, 0xc4201d4960, 0x0, 0xc4200c2101, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:341 +0x17d fp=0xc4204c7d38 sp=0xc4204c7c20 pc=0xfac8bd
github.com/hashicorp/nomad/client.(*FingerprintManager).watchDriver(0xc4201d2940, 0x219a6c0, 0xc4201d4960, 0x16c5a59, 0x4)
	/opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:295 +0x3d1 fp=0xc4204c7fb8 sp=0xc4204c7d38 pc=0xfac081
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc4204c7fc0 sp=0xc4204c7fb8 pc=0x45f071
created by github.com/hashicorp/nomad/client.(*FingerprintManager).setupDrivers
	/opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:192 +0x251

goroutine 1 [select, 240 minutes]:
github.com/hashicorp/nomad/command/agent.(*Command).handleSignals(0xc4201f6770, 0x1738010)
	/opt/gopath/src/github.com/hashicorp/nomad/command/agent/command.go:562 +0x22d
github.com/hashicorp/nomad/command/agent.(*Command).Run(0xc4201f6770, 0xc420098020, 0x5, 0x5, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/command/agent/command.go:551 +0xd45
github.com/hashicorp/nomad/vendor/github.com/mitchellh/cli.(*CLI).Run(0xc420461680, 0xc420461680, 0xc420447c00, 0x37)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/mitchellh/cli/cli.go:255 +0x1eb
main.RunCustom(0xc420098010, 0x6, 0x6, 0xc4204d9f60)
	/opt/gopath/src/github.com/hashicorp/nomad/main.go:127 +0x4c9
main.Run(0xc420098010, 0x6, 0x6, 0x12f3d54)
	/opt/gopath/src/github.com/hashicorp/nomad/main.go:72 +0x3f
main.main()
	/opt/gopath/src/github.com/hashicorp/nomad/main.go:68 +0x63

goroutine 19 [syscall, 240 minutes]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:131 +0xa6
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 72 [select, 240 minutes]:
github.com/hashicorp/nomad/command/agent/consul.(*checkWatcher).Run(0xc4201d2240, 0x2191400, 0xc4204fc240)
	/opt/gopath/src/github.com/hashicorp/nomad/command/agent/consul/check_watcher.go:186 +0x297
created by github.com/hashicorp/nomad/command/agent/consul.(*ServiceClient).Run
	/opt/gopath/src/github.com/hashicorp/nomad/command/agent/consul/client.go:300 +0x33a

goroutine 114 [semacquire]:
sync.runtime_SemacquireMutex(0xc4201d2954, 0x0)
	/usr/local/go/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0xc4201d2950)
	/usr/local/go/src/sync/mutex.go:134 +0xee
github.com/hashicorp/nomad/client.(*FingerprintManager).fingerprintDriver(0xc4201d2940, 0x16c4eef, 0x3, 0x7f6af40c2268, 0xc4201d4b40, 0x0, 0xc42004a000, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:327 +0x6e
github.com/hashicorp/nomad/client.(*FingerprintManager).watchDriver(0xc4201d2940, 0x219a8c0, 0xc4201d4b40, 0x16c4eef, 0x3)
	/opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:295 +0x3d1
created by github.com/hashicorp/nomad/client.(*FingerprintManager).setupDrivers
	/opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:192 +0x251

goroutine 133 [select]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).send(0xc4200ae9a0)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:392 +0x2ca
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.newSession
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:109 +0x406

goroutine 252 [select, 238 minutes]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Stream).Read(0xc4202e4540, 0xc420392000, 0x2000, 0x2000, 0x0, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/stream.go:133 +0x342
io/ioutil.devNull.ReadFrom(0x0, 0x21833c0, 0xc4202e4540, 0x14db760, 0x1, 0x7f6af07bd0d0)
	/usr/local/go/src/io/ioutil/ioutil.go:144 +0x92
io/ioutil.(*devNull).ReadFrom(0x22544e0, 0x21833c0, 0xc4202e4540, 0x7f6af07bd0d0, 0x22544e0, 0x1)
	<autogenerated>:1 +0x50
io.copyBuffer(0x21867c0, 0x22544e0, 0x21833c0, 0xc4202e4540, 0x0, 0x0, 0x0, 0xc4202c77b0, 0x0, 0x0)
	/usr/local/go/src/io/io.go:386 +0x2bb
io.Copy(0x21867c0, 0x22544e0, 0x21833c0, 0xc4202e4540, 0x0, 0x0, 0xc4202c7790)
	/usr/local/go/src/io/io.go:362 +0x68
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.copyStream(0x16c8c34, 0x6, 0x21867c0, 0x22544e0, 0x21833c0, 0xc4202e4540)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/stream.go:15 +0x79
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*RPCClient).SyncStreams
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/rpc_client.go:109 +0x119

....

goroutine 52 [select, 240 minutes]:
github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics.(*InmemSignal).run(0xc4201d2180)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics/inmem_signal.go:64 +0xf8
created by github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics.NewInmemSignal
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics/inmem_signal.go:38 +0x162

goroutine 53 [sleep]:
time.Sleep(0x3b9aca00)
	/usr/local/go/src/runtime/time.go:65 +0x130
github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics.(*Metrics).collectStats(0xc420208210)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics/metrics.go:168 +0x2f
created by github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics.New
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics/start.go:75 +0x12b

goroutine 54 [select, 237 minutes]:
github.com/hashicorp/nomad/command/agent/consul.(*ServiceClient).Run(0xc420208420)
	/opt/gopath/src/github.com/hashicorp/nomad/command/agent/consul/client.go:310 +0x4fe
created by github.com/hashicorp/nomad/command/agent.(*Agent).setupConsul
	/opt/gopath/src/github.com/hashicorp/nomad/command/agent/agent.go:950 +0x145

goroutine 55 [select]:
github.com/hashicorp/nomad/helper/pool.(*ConnPool).reap(0xc420356f00)
	/opt/gopath/src/github.com/hashicorp/nomad/helper/pool/pool.go:439 +0x37b
created by github.com/hashicorp/nomad/helper/pool.NewPool
	/opt/gopath/src/github.com/hashicorp/nomad/helper/pool/pool.go:170 +0x179

goroutine 56 [select]:
github.com/hashicorp/nomad/client.(*Client).rpcConnListener(0xc42045c000)
	/opt/gopath/src/github.com/hashicorp/nomad/client/rpc.go:241 +0x127
created by github.com/hashicorp/nomad/client.(*Client).setupClientRpc
	/opt/gopath/src/github.com/hashicorp/nomad/client/rpc.go:221 +0x134

goroutine 74 [select, 4 minutes]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Stream).Read(0xc420500540, 0xc4208d1000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/stream.go:133 +0x342
bufio.(*Reader).fill(0xc42059fbc0)
	/usr/local/go/src/bufio/bufio.go:97 +0x11a
bufio.(*Reader).ReadByte(0xc42059fbc0, 0xc42001d900, 0xc42001d970, 0xc42001d900)
	/usr/local/go/src/bufio/bufio.go:239 +0x39
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-msgpack/codec.(*ioDecReader).readn1(0xc420ca74a0, 0xc42055b610)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-msgpack/codec/decode.go:90 +0x7a
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-msgpack/codec.(*msgpackDecDriver).initReadNext(0xc420afa520)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-msgpack/codec/msgpack.go:540 +0x45
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-msgpack/codec.(*Decoder).decode(0xc42059fc20, 0x13e3ce0, 0xc420ca7650)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-msgpack/codec/decode.go:635 +0x49
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-msgpack/codec.(*Decoder).Decode(0xc42059fc20, 0x13e3ce0, 0xc420ca7650, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-msgpack/codec/decode.go:630 +0x7b
github.com/hashicorp/nomad/vendor/github.com/hashicorp/net-rpc-msgpackrpc.(*MsgpackCodec).read(0xc420d95300, 0x13e3ce0, 0xc420ca7650, 0x1, 0xc420ca7650)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/net-rpc-msgpackrpc/codec.go:121 +0x57
github.com/hashicorp/nomad/vendor/github.com/hashicorp/net-rpc-msgpackrpc.(*MsgpackCodec).ReadResponseHeader(0xc420d95300, 0xc420ca7650, 0x15c8940, 0xc4201e4080)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/net-rpc-msgpackrpc/codec.go:74 +0x41
github.com/hashicorp/nomad/vendor/github.com/hashicorp/net-rpc-msgpackrpc.CallWithCodec(0x2191c80, 0xc420d95300, 0x16dc6d1, 0x14, 0x15c8940, 0xc4201e4080, 0x150a6a0, 0xc42007e570, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/net-rpc-msgpackrpc/client.go:29 +0x10d
github.com/hashicorp/nomad/helper/pool.(*ConnPool).RPC(0xc420356f00, 0x7ffd3f881ef5, 0x4, 0x218a240, 0xc420747080, 0x1, 0x16dc6d1, 0x14, 0x15c8940, 0xc4201e4080, ...)
	/opt/gopath/src/github.com/hashicorp/nomad/helper/pool/pool.go:422 +0x1a8
github.com/hashicorp/nomad/client.(*Client).RPC(0xc42045c000, 0x16dc6d1, 0x14, 0x15c8940, 0xc4201e4080, 0x150a6a0, 0xc42007e570, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/client/rpc.go:68 +0x136
github.com/hashicorp/nomad/client.(*Client).watchAllocations(0xc42045c000, 0xc4200a8360)
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:1619 +0x27c
created by github.com/hashicorp/nomad/client.(*Client).run
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:1280 +0x7c

goroutine 57 [select]:
github.com/hashicorp/nomad/client.(*AllocGarbageCollector).Run(0xc4201d4640)
	/opt/gopath/src/github.com/hashicorp/nomad/client/gc.go:91 +0x150
created by github.com/hashicorp/nomad/client.NewClient
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:255 +0x7cc

goroutine 60 [select]:
github.com/hashicorp/nomad/client.(*FingerprintManager).runFingerprint(0xc4201d2940, 0x2188740, 0xc4204ed560, 0x37e11d600, 0x16c810c, 0x6)
	/opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:212 +0x243
created by github.com/hashicorp/nomad/client.(*FingerprintManager).setupFingerprinters
	/opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:158 +0x216

goroutine 26408 [select, 15 minutes]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).AcceptStream(0xc42094c420, 0xd2eab1, 0x1, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:207 +0xf2
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).Accept(0xc42094c420, 0xc4202c37c0, 0xd3317d, 0xc420758b00, 0xc4203178d0)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:197 +0x2b
github.com/hashicorp/nomad/client.(*Client).listenConn(0xc42045c000, 0xc42094c420)
	/opt/gopath/src/github.com/hashicorp/nomad/client/rpc.go:258 +0x2f
created by github.com/hashicorp/nomad/client.(*Client).rpcConnListener
	/opt/gopath/src/github.com/hashicorp/nomad/client/rpc.go:249 +0x173

goroutine 64 [IO wait]:
internal/poll.runtime_pollWait(0x7f6af40b7eb0, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc42041cb98, 0x72, 0xffffffffffffff00, 0x2187440, 0x217f978)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc42041cb98, 0xc4203bf000, 0x1000, 0x1000)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc42041cb80, 0xc4203bf000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a
net.(*netFD).Read(0xc42041cb80, 0xc4203bf000, 0x1000, 0x1000, 0x42f99b, 0xc4203cca40, 0x45b0e0)
	/usr/local/go/src/net/fd_unix.go:202 +0x52
net.(*conn).Read(0xc420092770, 0xc4203bf000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:176 +0x6d
net/http.(*persistConn).Read(0xc42035d560, 0xc4203bf000, 0x1000, 0x1000, 0xc4203282b0, 0xc4202bf738, 0x459140)
	/usr/local/go/src/net/http/transport.go:1391 +0x140
bufio.(*Reader).fill(0xc420357d40)
	/usr/local/go/src/bufio/bufio.go:97 +0x11a
bufio.(*Reader).Peek(0xc420357d40, 0x1, 0x0, 0x0, 0x0, 0xc420723b60, 0x0)
	/usr/local/go/src/bufio/bufio.go:129 +0x3a
net/http.(*persistConn).readLoop(0xc42035d560)
	/usr/local/go/src/net/http/transport.go:1539 +0x185
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1186 +0xa2e

goroutine 65 [select]:
net/http.(*persistConn).writeLoop(0xc42035d560)
	/usr/local/go/src/net/http/transport.go:1759 +0x165
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1187 +0xa53

goroutine 251 [select, 238 minutes]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Stream).Read(0xc4202e4240, 0xc42038c000, 0x2000, 0x2000, 0x0, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/stream.go:133 +0x342
io/ioutil.devNull.ReadFrom(0x0, 0x21833c0, 0xc4202e4240, 0x14db760, 0x1, 0x7f6af07bd0d0)
	/usr/local/go/src/io/ioutil/ioutil.go:144 +0x92
io/ioutil.(*devNull).ReadFrom(0x22544e0, 0x21833c0, 0xc4202e4240, 0x7f6af07bd0d0, 0x22544e0, 0x1)
	<autogenerated>:1 +0x50
io.copyBuffer(0x21867c0, 0x22544e0, 0x21833c0, 0xc4202e4240, 0x0, 0x0, 0x0, 0xc4204af7b0, 0x0, 0x0)
	/usr/local/go/src/io/io.go:386 +0x2bb
io.Copy(0x21867c0, 0x22544e0, 0x21833c0, 0xc4202e4240, 0x0, 0x0, 0xc4204af790)
	/usr/local/go/src/io/io.go:362 +0x68
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.copyStream(0x16c8c3a, 0x6, 0x21867c0, 0x22544e0, 0x21833c0, 0xc4202e4240)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/stream.go:15 +0x79
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*RPCClient).SyncStreams
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/rpc_client.go:108 +0x9d

goroutine 250 [select, 238 minutes]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Stream).Read(0xc42056c180, 0xc42037e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/stream.go:133 +0x342
bufio.(*Reader).Read(0xc420510660, 0xc4204ee6f0, 0x1, 0x9, 0xc42057bca8, 0x42cc38, 0x3)
	/usr/local/go/src/bufio/bufio.go:213 +0x30b
io.ReadAtLeast(0x2181600, 0xc420510660, 0xc4204ee6f0, 0x1, 0x9, 0x1, 0xc42001d900, 0x1737740, 0xc42005ad80)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2181600, 0xc420510660, 0xc4204ee6f0, 0x1, 0x9, 0xc42057bd10, 0x40629b, 0x3)
	/usr/local/go/src/io/io.go:327 +0x58
encoding/gob.decodeUintReader(0x2181600, 0xc420510660, 0xc4204ee6f0, 0x9, 0x9, 0xc42057bd88, 0x42c579, 0xc400000008, 0xc420096100)
	/usr/local/go/src/encoding/gob/decode.go:120 +0x63
encoding/gob.(*Decoder).recvMessage(0xc4201e4e00, 0xc42057bda0)
	/usr/local/go/src/encoding/gob/decoder.go:80 +0x57
encoding/gob.(*Decoder).decodeTypeSequence(0xc4201e4e00, 0x173c100, 0xc4201e4e00)
	/usr/local/go/src/encoding/gob/decoder.go:144 +0x14c
encoding/gob.(*Decoder).DecodeValue(0xc4201e4e00, 0x13e3ce0, 0xc4204147b0, 0x16, 0x0, 0x0)
	/usr/local/go/src/encoding/gob/decoder.go:212 +0xe6
encoding/gob.(*Decoder).Decode(0xc4201e4e00, 0x13e3ce0, 0xc4204147b0, 0x0, 0x0)
	/usr/local/go/src/encoding/gob/decoder.go:189 +0x16d
net/rpc.(*gobClientCodec).ReadResponseHeader(0xc4204edb00, 0xc4204147b0, 0xc4204ee6fc, 0x0)
	/usr/local/go/src/net/rpc/client.go:223 +0x45
net/rpc.(*Client).input(0xc420510720)
	/usr/local/go/src/net/rpc/client.go:109 +0xb5
created by net/rpc.NewClientWithCodec
	/usr/local/go/src/net/rpc/client.go:201 +0xc5

goroutine 87 [select]:
github.com/hashicorp/nomad/client.(*FingerprintManager).runFingerprint(0xc4201d2940, 0x2188a00, 0xc420361be0, 0x37e11d600, 0x16c756f, 0x5)
	/opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:212 +0x243
created by github.com/hashicorp/nomad/client.(*FingerprintManager).setupFingerprinters
	/opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:158 +0x216

goroutine 249 [select, 238 minutes]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).AcceptStream(0xc4200ae9a0, 0x0, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:207 +0xf2
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*MuxBroker).Run(0xc420588340)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/mux_broker.go:139 +0x33
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.NewRPCClient
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/rpc_client.go:88 +0x22f

goroutine 98 [semacquire]:
sync.runtime_SemacquireMutex(0xc4201d2954, 0x1416900)
	/usr/local/go/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0xc4201d2950)
	/usr/local/go/src/sync/mutex.go:134 +0xee
github.com/hashicorp/nomad/client.(*FingerprintManager).setNode(0xc4201d2940, 0xc42035cb40)
	/opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:54 +0x3a
github.com/hashicorp/nomad/client.(*FingerprintManager).fingerprintDriver(0xc4201d2940, 0x16c822c, 0x6, 0x7f6af40c1f98, 0xc4204f6780, 0x1, 0xc4202c8f00, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:380 +0x382
github.com/hashicorp/nomad/client.(*FingerprintManager).watchDriver(0xc4201d2940, 0x219a640, 0xc4204f6780, 0x16c822c, 0x6)
	/opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:295 +0x3d1
created by github.com/hashicorp/nomad/client.(*FingerprintManager).setupDrivers
	/opt/gopath/src/github.com/hashicorp/nomad/client/fingerprint_manager.go:192 +0x251

goroutine 248 [select, 238 minutes]:
github.com/hashicorp/nomad/client.(*AllocRunner).Run(0xc42051d980)
	/opt/gopath/src/github.com/hashicorp/nomad/client/alloc_runner.go:938 +0xcdc
created by github.com/hashicorp/nomad/client.(*Client).addAlloc
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:1912 +0x3e7

goroutine 121 [select]:
github.com/hashicorp/nomad/client.(*Client).registerAndHeartbeat(0xc42045c000)
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:1175 +0x1d1
created by github.com/hashicorp/nomad/client.NewClient
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:318 +0xed3

goroutine 122 [select, 1 minutes]:
github.com/hashicorp/nomad/client.(*Client).periodicSnapshot(0xc42045c000)
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:1263 +0x103
created by github.com/hashicorp/nomad/client.NewClient
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:321 +0xef8

goroutine 123 [select]:
github.com/hashicorp/nomad/client.(*Client).allocSync(0xc42045c000)
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:1531 +0x209
created by github.com/hashicorp/nomad/client.NewClient
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:324 +0xf1d

goroutine 124 [select, 4 minutes]:
github.com/hashicorp/nomad/client.(*Client).run(0xc42045c000)
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:1283 +0x153
created by github.com/hashicorp/nomad/client.NewClient
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:327 +0xf42

goroutine 125 [select]:
github.com/hashicorp/nomad/client.(*Client).emitStats(0xc42045c000)
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:2183 +0x238
created by github.com/hashicorp/nomad/client.NewClient
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:330 +0xf67

goroutine 126 [IO wait]:
internal/poll.runtime_pollWait(0x7f6af40b7c70, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc420205098, 0x72, 0xc4204c6c00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc420205098, 0xffffffffffffff00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc420205080, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:335 +0x1e2
net.(*netFD).accept(0xc420205080, 0xc4204c6e40, 0x4020c7, 0xc4202e2300)
	/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc4201da280, 0x715e58, 0x45b7b0, 0xc4204c6e88)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc4201da280, 0x173b8f0, 0xc4202e2280, 0x21914c0, 0xc42007f770)
	/usr/local/go/src/net/tcpsock.go:247 +0x49
net/http.(*Server).Serve(0xc420514000, 0x218f800, 0xc4201da280, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2695 +0x1b2
net/http.Serve(0x218f800, 0xc4201da280, 0x2186940, 0xc42007f500, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2323 +0x73
github.com/hashicorp/nomad/command/agent.NewHTTPServer.func1(0xc4201d2d40, 0x218f800, 0xc4201da280, 0xc4202eda80, 0xc4204f8f90)
	/opt/gopath/src/github.com/hashicorp/nomad/command/agent/http.go:111 +0x82
created by github.com/hashicorp/nomad/command/agent.NewHTTPServer
	/opt/gopath/src/github.com/hashicorp/nomad/command/agent/http.go:109 +0x40f

goroutine 25645 [IO wait]:
internal/poll.runtime_pollWait(0x7f6af40b78b0, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc420c38218, 0x72, 0xffffffffffffff00, 0x2187440, 0x217f978)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc420c38218, 0xc420636000, 0x1000, 0x1000)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc420c38200, 0xc420636000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a
net.(*netFD).Read(0xc420c38200, 0xc420636000, 0x1000, 0x1000, 0x18, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x52
net.(*conn).Read(0xc42051aac8, 0xc420636000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:176 +0x6d
bufio.(*Reader).Read(0xc4204113e0, 0xc4203b74c0, 0xc, 0xc, 0x40f2c0, 0xc420a5c180, 0xc420c992f8)
	/usr/local/go/src/bufio/bufio.go:213 +0x30b
io.ReadAtLeast(0x2181600, 0xc4204113e0, 0xc4203b74c0, 0xc, 0xc, 0xc, 0x2e0000000c, 0xc42050dce0, 0xc420759384)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2181600, 0xc4204113e0, 0xc4203b74c0, 0xc, 0xc, 0x0, 0x0, 0x0)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).recvLoop(0xc420759340, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:451 +0xfc
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).recv(0xc420759340)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:430 +0x2b
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.newSession
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:108 +0x3e4

goroutine 75 [select, 239 minutes]:
github.com/hashicorp/nomad/client.(*Client).watchNodeUpdates(0xc42045c000)
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:1774 +0x195
created by github.com/hashicorp/nomad/client.(*Client).registerAndHeartbeat
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:1159 +0x68

goroutine 76 [select, 240 minutes]:
github.com/hashicorp/nomad/client.(*Client).watchNodeEvents(0xc42045c000)
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:1328 +0x1f4
created by github.com/hashicorp/nomad/client.(*Client).registerAndHeartbeat
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:1162 +0x8d

goroutine 26407 [select]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).keepalive(0xc42094c420)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:308 +0x102
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.newSession
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:111 +0x455

goroutine 27585 [IO wait]:
internal/poll.runtime_pollWait(0x7f6af3ff01e8, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc4209dad18, 0x72, 0xffffffffffffff00, 0x2187440, 0x217f978)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc4209dad18, 0xc4209a3000, 0x1000, 0x1000)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc4209dad00, 0xc4209a3000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a
net.(*netFD).Read(0xc4209dad00, 0xc4209a3000, 0x1000, 0x1000, 0xc, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x52
net.(*conn).Read(0xc4205ca988, 0xc4209a3000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:176 +0x6d
bufio.(*Reader).Read(0xc42081dda0, 0xc420a4f2a0, 0xc, 0xc, 0x436f7b, 0xc420780e98, 0xd2fe74)
	/usr/local/go/src/bufio/bufio.go:213 +0x30b
io.ReadAtLeast(0x2181600, 0xc42081dda0, 0xc420a4f2a0, 0xc, 0xc, 0xc, 0xc, 0x0, 0x0)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2181600, 0xc42081dda0, 0xc420a4f2a0, 0xc, 0xc, 0x0, 0x0, 0x0)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).recvLoop(0xc420c89550, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:451 +0xfc
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).recv(0xc420c89550)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:430 +0x2b
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.newSession
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:108 +0x3e4

goroutine 27386 [select]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).send(0xc420ad2000)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:392 +0x2ca
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.newSession
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:109 +0x406

goroutine 27650 [select]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).send(0xc420c89550)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:392 +0x2ca
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.newSession
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:109 +0x406

goroutine 132 [IO wait]:
internal/poll.runtime_pollWait(0x7f6af40b7430, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc4205b0618, 0x72, 0xffffffffffffff00, 0x2187440, 0x217f978)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc4205b0618, 0xc4205f0000, 0x1000, 0x1000)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc4205b0600, 0xc4205f0000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a
net.(*netFD).Read(0xc4205b0600, 0xc4205f0000, 0x1000, 0x1000, 0xe1, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x52
net.(*conn).Read(0xc4201da070, 0xc4205f0000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:176 +0x6d
bufio.(*Reader).Read(0xc42059e4e0, 0xc4204ee6a0, 0xc, 0xc, 0xc, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:213 +0x30b
io.ReadAtLeast(0x2181600, 0xc42059e4e0, 0xc4204ee6a0, 0xc, 0xc, 0xc, 0xc, 0x0, 0x0)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2181600, 0xc42059e4e0, 0xc4204ee6a0, 0xc, 0xc, 0x0, 0x0, 0x0)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).recvLoop(0xc4200ae9a0, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:451 +0xfc
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).recv(0xc4200ae9a0)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:430 +0x2b
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.newSession
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:108 +0x3e4

goroutine 245 [select, 238 minutes]:
github.com/hashicorp/nomad/client.(*AllocRunner).Run(0xc42051d380)
	/opt/gopath/src/github.com/hashicorp/nomad/client/alloc_runner.go:938 +0xcdc
created by github.com/hashicorp/nomad/client.(*Client).addAlloc
	/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:1912 +0x3e7

goroutine 131 [chan receive, 238 minutes]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*Client).Start.func5.1(0xc4200c2660)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/client.go:570 +0x6d
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*Client).Start.func5
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/client.go:569 +0x3f

goroutine 229 [select, 238 minutes]:
github.com/hashicorp/nomad/client.(*AllocRunner).dirtySyncState(0xc42051d380)
	/opt/gopath/src/github.com/hashicorp/nomad/client/alloc_runner.go:642 +0xfe
created by github.com/hashicorp/nomad/client.(*AllocRunner).Run
	/opt/gopath/src/github.com/hashicorp/nomad/client/alloc_runner.go:834 +0x288

goroutine 231 [select, 238 minutes]:
github.com/hashicorp/nomad/client.(*TaskRunner).run(0xc420186000)
	/opt/gopath/src/github.com/hashicorp/nomad/client/task_runner.go:1107 +0x2cc
github.com/hashicorp/nomad/client.(*TaskRunner).Run(0xc420186000)
	/opt/gopath/src/github.com/hashicorp/nomad/client/task_runner.go:624 +0x6fc
created by github.com/hashicorp/nomad/client.(*AllocRunner).Run
	/opt/gopath/src/github.com/hashicorp/nomad/client/alloc_runner.go:927 +0xaed

Job file

N/A

The text was updated successfully, but these errors were encountered:

ivanprostran · 2018-04-15T13:26:02Z

I would appreciate any help regarding this issue.
Thank you!

chelseakomlo · 2018-04-16T13:08:32Z

Thanks for reporting this- we will be working on a fix today.

dadgar · 2018-04-16T18:01:26Z

@ivanprostran Can you share the full stack dump and a bit of the client logs before the panic?

ivanprostran · 2018-04-17T12:36:37Z

Hi, sorry for the late response.
File attached (just search for the following string "fatal error: concurrent map read and map write")
Thanks !!
Best regards,
Ivan Prostran

nomad-client.log

github-actions · 2022-12-01T02:28:12Z

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

chelseakomlo self-assigned this Apr 16, 2018

chelseakomlo added theme/client type/bug labels Apr 16, 2018

dadgar mentioned this issue Apr 16, 2018

Fixes races accessing node and updating it during fingerprinting #4162

Closed

dadgar mentioned this issue Apr 17, 2018

Fixes races accessing node and updating it during fingerprinting #4166

Merged

dadgar closed this as completed in #4166 Apr 17, 2018

github-actions bot locked as resolved and limited conversation to collaborators Dec 1, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nomad Client: fatal error: concurrent map read and map write #4154

Nomad Client: fatal error: concurrent map read and map write #4154

ivanprostran commented Apr 15, 2018 •

edited by dadgar

Loading

ivanprostran commented Apr 15, 2018

chelseakomlo commented Apr 16, 2018

dadgar commented Apr 16, 2018 •

edited

Loading

ivanprostran commented Apr 17, 2018

github-actions bot commented Dec 1, 2022

Nomad Client: fatal error: concurrent map read and map write #4154

Nomad Client: fatal error: concurrent map read and map write #4154

Comments

ivanprostran commented Apr 15, 2018 • edited by dadgar Loading

Nomad version

Operating system and Environment details

Issue

Reproduction steps

Nomad Server logs

Nomad Client logs

Job file

ivanprostran commented Apr 15, 2018

chelseakomlo commented Apr 16, 2018

dadgar commented Apr 16, 2018 • edited Loading

ivanprostran commented Apr 17, 2018

github-actions bot commented Dec 1, 2022

ivanprostran commented Apr 15, 2018 •

edited by dadgar

Loading

dadgar commented Apr 16, 2018 •

edited

Loading