Incorrect counting of map keys

[masterjg]

Terraform Version

Terraform v0.12.0-dev20190415H16
+ provider.aws v1.60.0-dev20190216h00-dev
+ provider.null v2.2.0-dev20190415h16-dev
+ provider.random v2.2.0-dev20190415h16-dev
+ provider.template v2.2.0-dev20190415h16-dev
+ provider.tls v2.0.0-dev20190415h16-dev

Terraform Configuration Files

module "db_backups" {
    ...
    user_access = {
        module.jenkins.resource_data["username"] = [
          "s3:*"
        ]
        module.developer.resource_data["username"] = [
          "s3:GetObject",
          "s3:ListBucket",
        ]
    }
}

(in module)
locals {
    users = keys(var.user_access)
}

data "aws_iam_user" "bucket" {
    count = length(local.users)
    user_name = local.users[count.index]
}

Debug Output

Crash Output

Expected Behavior

Terraform should easily detect that there are actually TWO users.

Actual Behavior

The "count" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the count depends on.

Steps to Reproduce

1. terraform init
2. terraform apply

Additional Context

References

[mildwonkey]

Hi @masterjg !
I'm sorry you've come across this issue; thank you for reporting it.

It would help if we could see a bit more of your configuration - I'm curious about the modules referenced in the user_access block:

    user_access = {
        module.jenkins.resource_data["username"] = [
          "s3:*"
        ]
        module.developer.resource_data["username"] = [
          "s3:GetObject",
          "s3:ListBucket",
        ]
    }

I'd like to know if either of those modules use count for the resource in question. You might be seeing that error if it's possible that either of those have a count of 0.

I was unable to reproduce this issue using hard-coded usernames, which is what led me to question the other modules:

module "db_backups" {
    user_access = {
        user_a = [
          "s3:*"
        ]
        user_b = [
          "s3:GetObject",
          "s3:ListBucket",
        ]
    }
}

[masterjg]

Yes of course...

output "resource_data" {
  value = {
    username = aws_iam_user.user.name
  }
}

resource "aws_iam_user" "user" {
  name = var.name
}

variable "name" {
  type = string
}

[masterjg]

BTW If I hardcode them like you do then everything works as expected. But for some reason terraform doesnt work with outputs from other modules here...

[apparentlymart]

This symptom suggests that the username values themselves are appearing as "unknown" during planning. I'm not sure exactly why that is -- is var.name here populated from a resource output too? -- but if they are unknown then this behavior is unfortunately correct, albeit confusing:

If both of those usernames are unknown during plan then Terraform can't tell whether they'll have distinct values once they are known (during apply) and so it can't tell if there will be one or two elements in that map. Conservatively then, it returns an unknown value from the keys function, which in turn causes an unknown value from the length function, and thus causes count to be unknown.

I think this could be worked around by making that a list instead of a map and putting the username in the value, since then the number of elements won't depend on what those names turn out to be at apply time:

module "db_backups" {
  ...
  user_access = [
    {
      username = module.jenkins.resource_data["username"]
      access = [
        "s3:*"
      ]
    },
    {
      username = module.developer.resource_data["username"]
      access = [
        "s3:GetObject",
        "s3:ListBucket",
      ]
    },
  ]
}

locals {
  users = var.user_access.*.username
}

data "aws_iam_user" "bucket" {
    count = length(local.users)
    user_name = local.users[count.index]
}

During plan in this case, local.users can be [<unknown>, <unknown>] and thus length can see that there are two elements even though their values aren't known yet.

[masterjg]

var.name is always known (it isn't output from another module). For example I can set variables to jenkins and developer

module "jenkins" {
...
    name = "jenkins"
}

module "developer" {
...
    name = "developer"
}

So I think terraform should know that there are only two of them?

[apparentlymart]

Thanks for the extra context, @masterjg!

In that case, I think the question here is: why does Terraform consider these names to be unknown when they are ultimately derived from a constant value?

There's a lot of indirection here, so I think to puzzle this out will require us to find a small, complete reproduction case and then follow the data flow to understand which expression evaluation is producing an unknown value first. It seems like one of the evaluation steps along the way is being too conservative, causing an unknown value where one isn't warranted.

[masterjg]

Here you go :)

incorrect-terraform-count.zip

[apparentlymart]

Thanks for the reproduction case, @masterjg! It was very helpful in tracing what's going on here.

This error is being produced during the refresh walk, which during initial create happens with a totally empty state. When Terraform constructs the graph for a refresh walk, it doesn't include any resource nodes unless they are already present in the state, under the presumption that there's therefore nothing to refresh, and so our refresh graph for this configuration is quite minimal:

The crucial detail here is that neither module.developer.aws_iam_user.user nor module.jenkins.aws_iam_user.user are in this graph because they don't yet exist in the state, and thus when we walk to module.db_backups.data.aws_iam_user.bucket and try to evaluate the count, these resources still don't exist in the state.

Once a resource has been processed during the plan phase, there is an entry in the state for the resource as a whole and for each of its potentially-multiple instances (if count is set). The expression evaluator uses that record in the state to decide whether the expression aws_iam_user.user should return a single object or a list of objects.

The problem here is that during the refresh phase, where we have no record of these in the state, the expression evaluator doesn't know whether aws_iam_user.user will be a single object or a list, and so it just returns an unknown value expecting this to be decided later. Normally that's fine, but in this particular situation it fails because count values cannot be unknown and the count for a data resource is evaluated during the refresh walk, too early for the resource to have been planned.

---

This is another problem in the family described by #17034, and can be fixed properly in the long term by moving the data source reads into the plan phase rather than doing them during the refresh phase, as that issue proposes. That's too big a change to make before the v0.12.0 release though, so tomorrow (since my work day is over now) I'm going to evaluate the following approach to see if it's practical as a short-term fix for this:

If count evaluates to an unknown value during the refresh walk, treat that similarly to when the data resource config itself contains unknown values and skip that data resource altogether. The subsequent plan walk should then try evaluating count again after the resources have been planned and thus get a known count result, and thus produce a plan to refresh these data sources during the apply step instead. That should then allow the full plan/apply cycle to complete and subsequent runs to behave as expected, modulo the existing limitations described in #17034 that were true before v0.12 too.

[masterjg]

Ok thanks for the update, @apparentlymart ! Hopefully this will be fixed one way or another soon enough and find its way to the next beta or dev snapshot :)

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.