fatal error: concurrent map read and map write

[elprans]

Terraform Version

Terraform v0.15.1
on linux_amd64
+ provider registry.terraform.io/hashicorp/local v2.1.0
+ provider registry.terraform.io/hashicorp/nomad v1.4.14
+ provider registry.terraform.io/hashicorp/null v3.1.0
+ provider registry.terraform.io/hashicorp/template v2.2.0
+ provider registry.terraform.local/dmacvicar/libvirt v0.6.3

Nomad Version

Name           Address      Port  Status  Leader  Protocol  Build        Datacenter  Region
nomad0.global  10.17.3.73   4648  alive   true    2         1.1.0-beta1  local-1     global
nomad1.global  10.17.3.104  4648  alive   false   2         1.1.0-beta1  local-1     global
nomad2.global  10.17.3.238  4648  alive   false   2         1.1.0-beta1  local-1     global

Provider Configuration

provider "nomad" {
  address = "http://nomad.service.${local.datacenter}.consul:4646"
}

Environment Variables

$ env | grep NOMAD_
NOMAD_ADDR=http://nomad.service.local-1.consul:4646

Affected Resource(s)

• nomad_job

Panic Output

PANIC OUTPUT

module.nomad-ebs.data.nomad_plugin.ebs: Reading...
╷
│ Error: Plugin did not respond
│ 
│   with nomad_job.loki,
│   on main.tf line 99, in resource "nomad_job" "loki":
│   99: resource "nomad_job" "loki" {
│ 
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).PlanResourceChange call. The plugin logs may contain more details.
╵
╷
│ Error: Plugin did not respond
│ 
│   with nomad_job.grafana,
│   on main.tf line 114, in resource "nomad_job" "grafana":
│  114: resource "nomad_job" "grafana" {
│ 
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).PlanResourceChange call. The plugin logs may contain more details.
╵
╷
│ Error: Plugin did not respond
│ 
│   with module.nomad-ebs.nomad_job.ebs_controller,
│   on ../modules/nomad-ebs/nomad-ebs.tf line 18, in resource "nomad_job" "ebs_controller":
│   18: resource "nomad_job" "ebs_controller" {
│ 
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).PlanResourceChange call. The plugin logs may contain more details.
╵
╷
│ Error: Plugin did not respond
│ 
│   with module.nomad-ebs.data.nomad_plugin.ebs,
│   on ../modules/nomad-ebs/nomad-ebs.tf line 50, in data "nomad_plugin" "ebs":
│   50: data "nomad_plugin" "ebs" {
│ 
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ReadDataSource call. The plugin logs may contain more details.
╵

Stack trace from the terraform-provider-nomad_v1.4.14_x4 plugin:

fatal error: concurrent map read and map write

goroutine 145 [running]:
runtime.throw(0x13cbf97, 0x21)
        /opt/goenv/versions/1.14.0/src/runtime/panic.go:1112 +0x72 fp=0xc000c9ead8 sp=0xc000c9eaa8 pc=0x432dd2
runtime.mapaccess1_faststr(0x12cbae0, 0xc0005c4450, 0x13aae8c, 0xa, 0x0)
        /opt/goenv/versions/1.14.0/src/runtime/map_faststr.go:21 +0x43c fp=0xc000c9eb48 sp=0xc000c9ead8 pc=0x41266c
net/textproto.MIMEHeader.Get(0xc0005c4450, 0x13aae8c, 0xa, 0x0, 0x0)
        /opt/goenv/versions/1.14.0/src/net/textproto/header.go:34 +0x5d fp=0xc000c9eb80 sp=0xc000c9eb48 pc=0x79167d
net/http.Header.Get(...)
        /opt/goenv/versions/1.14.0/src/net/http/header.go:46
net/http.(*Request).requiresHTTP1(0xc0002f4100, 0x0)
        /opt/goenv/versions/1.14.0/src/net/http/request.go:1441 +0x48 fp=0xc000c9ebb8 sp=0xc000c9eb80 pc=0x7e79b8
net/http.(*Transport).connectMethodForRequest(0xc0005cdb80, 0xc0005ac990, 0x0, 0xc0005e60c0, 0x4, 0xc0003a6090, 0x21, 0x0, 0x1, 0x1320)
        /opt/goenv/versions/1.14.0/src/net/http/transport.go:755 +0xa3 fp=0xc000c9ebf8 sp=0xc000c9ebb8 pc=0x7f82b3
net/http.(*Transport).roundTrip(0xc0005cdb80, 0xc0002f4100, 0xc0011138e0, 0xc000c9ee88, 0x40c928)
        /opt/goenv/versions/1.14.0/src/net/http/transport.go:542 +0x6a7 fp=0xc000c9ee40 sp=0xc000c9ebf8 pc=0x7f6e07
net/http.(*Transport).RoundTrip(0xc0005cdb80, 0xc0002f4100, 0xc0005cdb80, 0x0, 0x0)
        /opt/goenv/versions/1.14.0/src/net/http/roundtrip.go:17 +0x35 fp=0xc000c9ee78 sp=0xc000c9ee40 pc=0x7e90e5
net/http.send(0xc0002f4100, 0x1700340, 0xc0005cdb80, 0x0, 0x0, 0x0, 0xc00035c358, 0xc000c9f048, 0x1, 0x0)
        /opt/goenv/versions/1.14.0/src/net/http/client.go:252 +0x43e fp=0xc000c9efe0 sp=0xc000c9ee78 pc=0x7bfbfe
net/http.(*Client).send(0xc0005c4540, 0xc0002f4100, 0x0, 0x0, 0x0, 0xc00035c358, 0x0, 0x1, 0x0)
        /opt/goenv/versions/1.14.0/src/net/http/client.go:176 +0xfa fp=0xc000c9f060 sp=0xc000c9efe0 pc=0x7bf61a
net/http.(*Client).do(0xc0005c4540, 0xc0002f4100, 0x0, 0x0, 0x0)
        /opt/goenv/versions/1.14.0/src/net/http/client.go:699 +0x44a fp=0xc000c9f238 sp=0xc000c9f060 pc=0x7c12ea
net/http.(*Client).Do(...)
        /opt/goenv/versions/1.14.0/src/net/http/client.go:567
github.com/hashicorp/nomad/api.(*Client).doRequest(0xc0005b8c40, 0xc00016c070, 0x3, 0xc0004960a0, 0x11, 0xc00016c070)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/nomad/[email protected]/api.go:714 +0xa0 fp=0xc000c9f2d8 sp=0xc000c9f238 pc=0xf08c60
github.com/hashicorp/nomad/api.(*Client).write(0xc0005b8c40, 0xc0004960a0, 0x11, 0x112fba0, 0xc0005d6880, 0x112fbe0, 0xc0011157a0, 0x0, 0x0, 0x0, ...)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/nomad/[email protected]/api.go:884 +0x1a6 fp=0xc000c9f398 sp=0xc000c9f2d8 pc=0xf0a016
github.com/hashicorp/nomad/api.(*Jobs).PlanOpts(0xc000c9f4b8, 0xc000581680, 0xc000c9f49e, 0x0, 0x1fc01c0, 0xc0005c1000, 0xc0005c10b0, 0xc000581680)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/nomad/[email protected]/jobs.go:369 +0x164 fp=0xc000c9f418 sp=0xc000c9f398 pc=0xf12594
github.com/hashicorp/terraform-provider-nomad/nomad.resourceJobCustomizeDiff(0xc000b8e680, 0x12d3760, 0xc000b93e20, 0xc00035e740, 0xc000b8e680)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-nomad/nomad/resource_job.go:600 +0x3e0 fp=0xc000c9f5b8 sp=0xc000c9f418 pc=0x10987c0
github.com/hashicorp/terraform-plugin-sdk/helper/schema.schemaMap.Diff(0xc0002c6ff0, 0xc0008cc2d0, 0xc00059ff20, 0x1401e08, 0x12d3760, 0xc000b93e20, 0x1720c00, 0x0, 0xffffffffffffffff, 0x0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/[email protected]/helper/schema/schema.go:524 +0xac2 fp=0xc000c9f788 sp=0xc000c9f5b8 pc=0xebfbb2
github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Resource).simpleDiff(0xc0002d2cf0, 0xc0008cc2d0, 0xc00059ff20, 0x12d3760, 0xc000b93e20, 0xc00059ff01, 0xc0004b58b0, 0x40d59d)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/[email protected]/helper/schema/resource.go:356 +0x85 fp=0xc000c9f860 sp=0xc000c9f788 pc=0xeb3ef5
github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Provider).SimpleDiff(0xc0001ead00, 0xc0004b5a98, 0xc0008cc2d0, 0xc00059ff20, 0xc00059e1e0, 0xc00059ff20, 0x0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/[email protected]/helper/schema/provider.go:321 +0x99 fp=0xc000c9f8c0 sp=0xc000c9f860 pc=0xeb1f39
github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin.(*GRPCProviderServer).PlanResourceChange(0xc00000f298, 0x171f740, 0xc00059e180, 0xc0003e81c0, 0xc00000f298, 0xc00059e180, 0xc0005d3ba0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/[email protected]/internal/helper/plugin/grpc_provider.go:633 +0x79c fp=0xc000c9fb40 sp=0xc000c9f8c0 pc=0xed551c
github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5._Provider_PlanResourceChange_Handler(0x1359100, 0xc00000f298, 0x171f740, 0xc00059e180, 0xc0000fe420, 0x0, 0x171f740, 0xc00059e180, 0xc0005dc000, 0x1fe8)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/[email protected]/internal/tfplugin5/tfplugin5.pb.go:3287 +0x217 fp=0xc000c9fbb0 sp=0xc000c9fb40 pc=0xdc4ad7
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000280fc0, 0x172e760, 0xc0002f7c80, 0xc0005be000, 0xc0002c7830, 0x2039908, 0x0, 0x0, 0x0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/[email protected]/server.go:1171 +0x50a fp=0xc000c9fe40 sp=0xc000c9fbb0 pc=0x8da7ba
google.golang.org/grpc.(*Server).handleStream(0xc000280fc0, 0x172e760, 0xc0002f7c80, 0xc0005be000, 0x0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/[email protected]/server.go:1494 +0xccd fp=0xc000c9ff68 sp=0xc000c9fe40 pc=0x8de94d
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc0005822d0, 0xc000280fc0, 0x172e760, 0xc0002f7c80, 0xc0005be000)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/[email protected]/server.go:834 +0xa1 fp=0xc000c9ffb8 sp=0xc000c9ff68 pc=0x8ebfb1
runtime.goexit()
        /opt/goenv/versions/1.14.0/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000c9ffc0 sp=0xc000c9ffb8 pc=0x463091
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/[email protected]/server.go:832 +0x204

goroutine 1 [select, 5 minutes]:
github.com/hashicorp/go-plugin.Serve(0xc000995ed8)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/[email protected]/server.go:472 +0xa40
github.com/hashicorp/terraform-plugin-sdk/plugin.Serve(0xc00063ff50)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/[email protected]/plugin/serve.go:76 +0x1e7
main.main()
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-nomad/main.go:11 +0x60

goroutine 17 [select, 1 minutes]:
go.opencensus.io/stats/view.(*worker).start(0xc000150100)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/[email protected]/stats/view/worker.go:276 +0x100
created by go.opencensus.io/stats/view.init.0
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/[email protected]/stats/view/worker.go:34 +0x68

goroutine 73 [select, 5 minutes]:
github.com/hashicorp/go-plugin.(*gRPCBrokerServer).Recv(0xc0002c75c0, 0x0, 0x0, 0x0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/[email protected]/grpc_broker.go:121 +0xb3
github.com/hashicorp/go-plugin.(*GRPCBroker).Run(0xc000152140)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/[email protected]/grpc_broker.go:411 +0x7a
created by github.com/hashicorp/go-plugin.(*GRPCServer).Init
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/[email protected]/grpc_server.go:85 +0x398

goroutine 74 [IO wait, 5 minutes]:
internal/poll.runtime_pollWait(0x7efd29586e38, 0x72, 0xffffffffffffffff)
        /opt/goenv/versions/1.14.0/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc000814078, 0x72, 0x1001, 0x1000, 0xffffffffffffffff)
        /opt/goenv/versions/1.14.0/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
        /opt/goenv/versions/1.14.0/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000814060, 0xc00020f000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /opt/goenv/versions/1.14.0/src/internal/poll/fd_unix.go:169 +0x19b
os.(*File).read(...)
        /opt/goenv/versions/1.14.0/src/os/file_unix.go:263
os.(*File).Read(0xc00035c000, 0xc00020f000, 0x1000, 0x1000, 0x400, 0x11783c0, 0x1)
        /opt/goenv/versions/1.14.0/src/os/file.go:116 +0x71
bufio.(*Reader).Read(0xc0002f9f50, 0xc0001bec00, 0x400, 0x400, 0x0, 0x0, 0x0)
        /opt/goenv/versions/1.14.0/src/bufio/bufio.go:226 +0x24f
github.com/hashicorp/go-plugin.copyChan(0x1735e20, 0xc0008155c0, 0xc0000405a0, 0x1700600, 0xc00035c000)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/[email protected]/grpc_stdio.go:181 +0xc6
created by github.com/hashicorp/go-plugin.newGRPCStdioServer
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/[email protected]/grpc_stdio.go:37 +0xb1

goroutine 75 [IO wait, 5 minutes]:
internal/poll.runtime_pollWait(0x7efd29586c78, 0x72, 0xffffffffffffffff)
        /opt/goenv/versions/1.14.0/src/runtime/netpoll.go:203 +0x55

Error: The terraform-provider-nomad_v1.4.14_x4 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

[jrasell]

Thanks for the report @elprans .

I believe this is due a race condition in the Nomad API that started to happen when #203 was merged. Before that, we didn't modify the headers.

The API was fixed in hashicorp/nomad#10302 and we'll be picking up this change in our next release. I will close out this ticket, referencing #215 as the canonical source for tracking this bug. If you have any follow up questions, please do not hesitate to ask.