0.6.0-rc1: client: fatal error: concurrent map read/iteration and map write

[rlenk]

Nomad version

Nomad v0.6.0-rc1 (5187fcf+CHANGES)

Operating system and Environment details

CentOS 7.3.1611
Docker 17.03.2-ce
Consul v0.8.5

Issue

Nomad clients fail with one of two errors:

1. fatal error: concurrent map read and map write
2. fatal error: concurrent map iteration and map write

Nomad Client logs (full log: nomad_map_panic.txt)

fatal error: concurrent map iteration and map write

goroutine 2301701 [running]:
runtime.throw(0x132c435, 0x26)
	/opt/go/src/runtime/panic.go:596 +0x95 fp=0xc42c2465b0 sp=0xc42c246590
runtime.mapiternext(0xc435af0480)
	/opt/go/src/runtime/hashmap.go:737 +0x7ee fp=0xc42c246660 sp=0xc42c2465b0
runtime.mapiterinit(0x1148740, 0xc4365866c0, 0xc435af0480)
	/opt/go/src/runtime/hashmap.go:727 +0x2b3 fp=0xc42c2466b8 sp=0xc42c246660
reflect.mapiterinit(0x1148740, 0xc4365866c0, 0x195)
	/opt/go/src/runtime/hashmap.go:1150 +0x54 fp=0xc42c2466e8 sp=0xc42c2466b8
reflect.Value.MapKeys(0x1148740, 0xc436586710, 0x195, 0x1244040, 0x101, 0xc424a8c8c8)
	/opt/go/src/reflect/value.go:1092 +0xd1 fp=0xc42c246798 sp=0xc42c2466e8
github.com/hashicorp/nomad/vendor/github.com/ugorji/go/codec.(*encFnInfo).kMap(0xc424a8c888, 0x1148740, 0xc436586710, 0x195)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/ugorji/go/codec/encode.go:704 +0x3d9 fp=0xc42c2468a8 sp=0xc42c246798
github.com/hashicorp/nomad/vendor/github.com/ugorji/go/codec.(*Encoder).doEncodeValue(0xc4338257c0, 0x1148740, 0xc436586710, 0x195, 0x0, 0x0, 0x101)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/ugorji/go/codec/encode.go:1222 +0x6d fp=0xc42c246910 sp=0xc42c2468a8
github.com/hashicorp/nomad/vendor/github.com/ugorji/go/codec.(*Encoder).encodeValue(0xc4338257c0, 0x1148740, 0xc436586710, 0x195, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/ugorji/go/codec/encode.go:1237 +0xa7 fp=0xc42c246968 sp=0xc42c246910
github.com/hashicorp/nomad/vendor/github.com/ugorji/go/codec.(*encFnInfo).kStruct(0xc424a8c808, 0x1208ca0, 0xc4365866f0, 0x199)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/ugorji/go/codec/encode.go:608 +0x723 fp=0xc42c246aa8 sp=0xc42c246968
github.com/hashicorp/nomad/vendor/github.com/ugorji/go/codec.(*Encoder).doEncodeValue(0xc4338257c0, 0x1208ca0, 0xc4365866f0, 0x199, 0x0, 0x0, 0x100)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/ugorji/go/codec/encode.go:1222 +0x6d fp=0xc42c246b10 sp=0xc42c246aa8
github.com/hashicorp/nomad/vendor/github.com/ugorji/go/codec.(*Encoder).encodeI(0xc4338257c0, 0x1236ee0, 0xc4365866f0, 0x100)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/ugorji/go/codec/encode.go:1230 +0xc9 fp=0xc42c246b68 sp=0xc42c246b10
github.com/hashicorp/nomad/vendor/github.com/ugorji/go/codec.(*Encoder).encode(0xc4338257c0, 0x1236ee0, 0xc4365866f0)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/ugorji/go/codec/encode.go:1166 +0x13e fp=0xc42c246bb0 sp=0xc42c246b68
github.com/hashicorp/nomad/vendor/github.com/ugorji/go/codec.(*Encoder).Encode(0xc4338257c0, 0x1236ee0, 0xc4365866f0, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/ugorji/go/codec/encode.go:1071 +0x84 fp=0xc42c246bd8 sp=0xc42c246bb0
github.com/hashicorp/nomad/client.putObject(0xc43414e680, 0x1b2c130, 0x9, 0x9, 0x1236ee0, 0xc4365866f0, 0xc42c246dd8, 0xa77f6f)
	/opt/gopath/src/github.com/hashicorp/nomad/client/state_database.go:35 +0xb5 fp=0xc42c246c90 sp=0xc42c246bd8
github.com/hashicorp/nomad/client.(*AllocRunner).saveAllocRunnerState.func1(0xc430cfb260, 0x1357638, 0xc42c246e18)
	/opt/gopath/src/github.com/hashicorp/nomad/client/alloc_runner.go:403 +0x69a fp=0xc42c246de0 sp=0xc42c246c90
github.com/hashicorp/nomad/vendor/github.com/boltdb/bolt.safelyCall(0xc433f6a4c0, 0xc430cfb260, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/boltdb/bolt/db.go:745 +0x75 fp=0xc42c246e08 sp=0xc42c246de0
github.com/hashicorp/nomad/vendor/github.com/boltdb/bolt.(*batch).run.func1(0xc430cfb260, 0x1357608, 0xc430cfb260)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/boltdb/bolt/db.go:694 +0x62 fp=0xc42c246e58 sp=0xc42c246e08
github.com/hashicorp/nomad/vendor/github.com/boltdb/bolt.(*DB).Update(0xc4201961c0, 0xc42c246f20, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/boltdb/bolt/db.go:573 +0x9f fp=0xc42c246ea8 sp=0xc42c246e58
github.com/hashicorp/nomad/vendor/github.com/boltdb/bolt.(*batch).run(0xc433f6a540)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/boltdb/bolt/db.go:700 +0x10b fp=0xc42c246f48 sp=0xc42c246ea8
github.com/hashicorp/nomad/vendor/github.com/boltdb/bolt.(*batch).(github.com/hashicorp/nomad/vendor/github.com/boltdb/bolt.run)-fm()
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/boltdb/bolt/db.go:674 +0x2a fp=0xc42c246f60 sp=0xc42c246f48
sync.(*Once).Do(0xc433f6a550, 0xc42436efa8)
	/opt/go/src/sync/once.go:44 +0xbe fp=0xc42c246f98 sp=0xc42c246f60
github.com/hashicorp/nomad/vendor/github.com/boltdb/bolt.(*batch).trigger(0xc433f6a540)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/boltdb/bolt/db.go:674 +0x55 fp=0xc42c246fc8 sp=0xc42c246f98
github.com/hashicorp/nomad/vendor/github.com/boltdb/bolt.(*batch).(github.com/hashicorp/nomad/vendor/github.com/boltdb/bolt.trigger)-fm()
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/boltdb/bolt/db.go:644 +0x2a fp=0xc42c246fe0 sp=0xc42c246fc8
runtime.goexit()
	/opt/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42c246fe8 sp=0xc42c246fe0
created by time.goFunc
	/opt/go/src/time/sleep.go:170 +0x44

[dadgar]

@rlenk Thanks for the report! Any reproduction steps?

[rlenk]

@dadgar I don't have any specific steps to reproduce the error, except to say that the cluster is under load. There are five servers, 20 clients, ~100 service jobs and ~300 batch jobs that run hourly. This error seems similar to #2668 that I opened against 0.5.6 back in May.

[dadgar]

@rlenk And from the build I gave you you never saw a problem? Will help narrow down possible commits that caused it!

[rlenk]

@dadgar No, I didn't see this issue with the 0.6.0dev build you provided. It appeared after installing -rc1

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.