Using for_each in a resource leads to runtime panic

[JanMa]

Hello,

I am trying to create multiple namespaces, ACL policies and tokens using the latest terraform provider for Nomad.
To simplify the configuration I am making use of the for_each stanza on multiple resources. When running terraform plan or terraform apply the provider crashes with a panic which can be seen below.

Using the -parallelism=1 flag it works without issues.

It seems to me that this issue was introduced in the release 1.4.14. If I pin the provider to version 1.4.13,1.4.12 or 1.4.11 everything works just fine.

Terraform Version

Terraform v0.15.0
on linux_amd64
+ provider registry.terraform.io/hashicorp/nomad v1.4.14

This command will also output the provider version, please include that as well.

Nomad Version

Name                Address    Port  Status  Leader  Protocol  Build  Datacenter  Region
my-pc.global        127.0.0.1  4648  alive   true    2         1.0.4  dc1         global

Provider Configuration

Which values are you setting in the provider configuration?

provider "nomad" {
  address   = "http://127.0.0.1:4646"
  secret_id = "<nomad bootstrap token here>"
}

Affected Resource(s)

Please list the resources as a list, for example:

• nomad_namespace
• nomad_acl_token
• nomad_acl_policy

possibly all available resources.

If this issue appears to affect multiple resources, it may be an issue with Terraform's core, so please mention this.

Terraform Configuration Files

provider "nomad" {
  address   = "http://127.0.0.1:4646"
  secret_id = "<nomad bootstrap token here>"
}

variable "namespaces" {
  type = set(any)
  default = [
    "team-01",
    "team-02",
    "team-03",
    "team-04",
    "team-05",
    "team-06",
    "team-07",
    "team-08",
    "team-09",
    "team-10",
    "team-11",
    "team-12",
    "team-13",
    "team-14",
    "team-15",
    "team-16",
    "team-17",
    "team-18",
    "team-19",
    "team-20",
  ]
}

resource "nomad_namespace" "namespaces" {
  for_each    = var.namespaces
  name        = each.key
  description = "Services of ${replace(each.key, "-", " ")}"
}

resource "nomad_acl_policy" "policies" {
  for_each    = var.namespaces
  name        = each.key
  description = "Submit jobs to namespace of ${replace(each.key, "-", " ")}"
  rules_hcl   = <<EOT
namespace "${each.key}" {
  policy = "read"
  capabilities = ["submit-job","dispatch-job","read-logs"]
}
namespace "default" {
  capabilities = ["list-jobs"]
}
node {
  policy = "read"
}
EOT
}

resource "nomad_acl_token" "tokens" {
  for_each = nomad_acl_policy.policies

  type = "client"
  name = each.key
  policies = [each.value.name]
}

Panic Output

Stack trace from the terraform-provider-nomad_v1.4.14_x4 plugin:                                                                                                                                                                               
                                                                                                                                                                                                                                               
fatal error: concurrent map writes                                                                                                                                                                                                             
fatal error: concurrent map writes                                                                                                                                                                                                             
                                                                                                                                                                                                                                               
goroutine 177 [running]:                                                                                                                                                                                                                       
runtime.throw(0x13b897e, 0x15)                                                                                                                                                                                                                 
        /opt/goenv/versions/1.14.0/src/runtime/panic.go:1112 +0x72 fp=0xc000853658 sp=0xc000853628 pc=0x432dd2                                                                                                                                 
runtime.mapassign_faststr(0x12cbae0, 0xc0006c9110, 0x13b01ff, 0xf, 0xc000650640)                                                                                                                                                               
        /opt/goenv/versions/1.14.0/src/runtime/map_faststr.go:291 +0x3de fp=0xc0008536c0 sp=0xc000853658 pc=0x412eee                                                                                                                           
net/textproto.MIMEHeader.Add(...)                                                                                                                                                                                                              
        /opt/goenv/versions/1.14.0/src/net/textproto/header.go:15                                                                                                                                                                              
net/http.Header.Add(...)                                                                                                                                                                                                                       
        /opt/goenv/versions/1.14.0/src/net/http/header.go:28                                                                                                                                                                                   
github.com/hashicorp/nomad/api.(*request).toHTTP(0xc000677110, 0xc000677178, 0xc0006bac90, 0xc0006bac60)                                                                                                                                       
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/nomad/[email protected]/api.go:629 +0x19a fp=0xc000853778 sp=0xc0008536c0 pc=0xf07d3a                                                      
github.com/hashicorp/nomad/api.(*Client).doRequest(0xc0002023f0, 0xc000677110, 0x3, 0xc000650640, 0x16, 0xc000677110)                                                                                                                          
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/nomad/[email protected]/api.go:709 +0x40 fp=0xc000853818 sp=0xc000853778 pc=0xf08c00                                                       
github.com/hashicorp/nomad/api.(*Client).query(0xc0002023f0, 0xc000650640, 0x16, 0x112e660, 0xc000653280, 0x0, 0x0, 0x0, 0x0)                                                                                                                  
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/nomad/[email protected]/api.go:833 +0xd3 fp=0xc0008538a0 sp=0xc000853818 pc=0xf099e3                                                       
github.com/hashicorp/nomad/api.(*ACLPolicies).Info(0xc000853980, 0xc00035a7d0, 0x7, 0x0, 0x1, 0x0, 0x2074630, 0x1404418)                                                                                                                       
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/nomad/[email protected]/acl.go:58 +0xd1 fp=0xc000853908 sp=0xc0008538a0 pc=0xf00181                                                        
github.com/hashicorp/terraform-provider-nomad/nomad.resourceACLPolicyRead(0xc0006770a0, 0x12d3760, 0xc000304900, 0xc0006770a0, 0x0)                                                                                                            
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-nomad/nomad/resource_acl_policy.go:113 +0x14a fp=0xc0008539c8 sp=0xc000853908 pc=0x109181a                                                       
github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Resource).RefreshWithoutUpgrade(0xc0002a94d0, 0xc000370c30, 0x12d3760, 0xc000304900, 0xc00000eda8, 0x0, 0x0)                                                                         
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/[email protected]/helper/schema/resource.go:460 +0x119 fp=0xc000853a38 sp=0xc0008539c8 pc=0xeb44c9                                                   
github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin.(*GRPCProviderServer).ReadResource(0xc0003088c0, 0x171f740, 0xc00082c150, 0xc0008380c0, 0xc0003088c0, 0x0, 0xc000848ba0)                                                      
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/[email protected]/internal/helper/plugin/grpc_provider.go:525 +0x3d8 fp=0xc000853b40 sp=0xc000853a38 pc=0xed45e8                                     
github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5._Provider_ReadResource_Handler(0x1359100, 0xc0003088c0, 0x171f740, 0xc00082c150, 0xc000838060, 0x0, 0x171f740, 0xc00082c150, 0xc00084a000, 0x12b)                                 
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/[email protected]/internal/tfplugin5/tfplugin5.pb.go:3269 +0x217 fp=0xc000853bb0 sp=0xc000853b40 pc=0xdc4847                                         
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000216e00, 0x172e760, 0xc000802c00, 0xc00083a000, 0xc0002c3020, 0x20398f0, 0x0, 0x0, 0x0)                                                                                                  
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/[email protected]/server.go:1171 +0x50a fp=0xc000853e40 sp=0xc000853bb0 pc=0x8da7ba                                                                                     
google.golang.org/grpc.(*Server).handleStream(0xc000216e00, 0x172e760, 0xc000802c00, 0xc00083a000, 0x0)                                                                                                                                        
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/[email protected]/server.go:1494 +0xccd fp=0xc000853f68 sp=0xc000853e40 pc=0x8de94d                                                                                     
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc000122170, 0xc000216e00, 0x172e760, 0xc000802c00, 0xc00083a000)                                                                                                                       
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/[email protected]/server.go:834 +0xa1 fp=0xc000853fb8 sp=0xc000853f68 pc=0x8ebfb1                                                                                       
runtime.goexit()                                                                                                                                                                                                                               
        /opt/goenv/versions/1.14.0/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000853fc0 sp=0xc000853fb8 pc=0x463091                                                                                                                               
created by google.golang.org/grpc.(*Server).serveStreams.func1                                                                                                                                                                                 
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/[email protected]/server.go:832 +0x204 
...

Expected Behavior

All resources should be created or their state should be refreshed

Actual Behavior

The provider crashed.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. start a nomad dev server with ACLs enabled using nomad agent -dev -acl-enabled
2. run nomad acl bootstrap and paste the token in the terraform provider config
3. run terraform init
4. run terraform plan or terraform apply

Important Factoids

• The setup is using ACLs

[lgfa29]

Thanks for the report @JanMa.

This is due a race condition in the Nomad API that started to happen when #203 was merged. Before that, we didn't modify the headers.

The API was fixed in hashicorp/nomad#10302 and we'll be picking up this change in our next release.