Required field using data source output isn't evaluated as set

[jacobbednarz]

While chasing down a customer issue, I discovered that a Required field (here cloudflare_logpush_job.example_job.ownership_challenge) pointing at a data source value doesn't register as being set despite the data source value definitely being present (confirmed using outputs). This results in the following error:

Error: "ownership_challenge": required field is not set

Versions

• Terraform v0.13.6
• provider registry.terraform.io/cloudflare/cloudflare v2.18.0 (https://github.com/cloudflare/terraform-provider-cloudflare)
• terraform-sdk-plugin v1.16.0

(example configuration from https://registry.terraform.io/providers/cloudflare/cloudflare/latest/docs/resources/logpush_job)

resource "cloudflare_logpush_ownership_challenge" "ownership_challenge" {
  zone_id = "d41d8cd98f00b204e9800998ecf8427e"
  destination_conf = "s3://my-bucket-path?region=us-west-2"
}

data "aws_s3_bucket_object" "challenge_file" {
  bucket = "my-bucket-path"
  key    = cloudflare_logpush_ownership_challenge.ownership_challenge.ownership_challenge_filename
}

resource "cloudflare_logpush_job" "example_job" {
  enabled = true
  zone_id = "d41d8cd98f00b204e9800998ecf8427e"
  name = "My-logpush-job"
  logpull_options = "fields=RayID,ClientIP,EdgeStartTimestamp&timestamps=rfc3339"
  destination_conf = "s3://my-bucket-path?region=us-west-2"
  ownership_challenge = data.aws_s3_bucket_object.challenge_file.body
  dataset = "http_requests"
}

I unfortunately haven't been able to narrow down a smaller reproduction test case however I suspect any two resources where one is a data source and the other has a required field may do it.

I've also tried using explicit depends_on in the resources to no avail.

cc @paddycarver from our discussion in Slack.

[ewbankkit]

In the sample configuration

ownership_challenge = aws_s3_bucket_object.challenge_file.body

should be

ownership_challenge = data.aws_s3_bucket_object.challenge_file.body

[jacobbednarz]

Thanks @ewbankkit, that was my bad when stripping down this test case. I'm afraid the reference isn't the issue here 😞 I've updated the test case accordingly.

[vital4ik]

I am getting the same issue on the tf 14.9 and cf provider 2.19.2
the apply fails immediately, I am trying to create a bucket first, then put data file into the bucket:

resource "google_storage_bucket" "log-storage" {
  project       = var.gcp-project
  name          = "${replace(var.global_zonename,".","-")}-logs"
  location      = "US"
  force_destroy = true

  lifecycle_rule {
    condition {
      age = 365
    }
    action {
      type = "SetStorageClass"
      storage_class = "COLDLINE"
    }
  }
}
resource "google_project_iam_binding" "cf-storageadmin" {
  project                                = var.gcp-project
  role                                   = "roles/storage.objectAdmin"

  members = [
                                         "serviceAccount:[email protected]",
  ]
}
data "google_iam_policy" "cf-storageadmin" {
  depends_on = [google_project_iam_binding.cf-storageadmin]
  binding {
    role = "roles/storage.objectAdmin"
    members = [
      "serviceAccount:[email protected]",
    ]
  }
}
resource "google_storage_bucket_iam_policy" "add-cf-storage-admin" {
  bucket = google_storage_bucket.log-storage.name
  policy_data = data.google_iam_policy.cf-storageadmin.policy_data
}
resource "cloudflare_logpush_ownership_challenge" "ownership_challenge" {
  depends_on = [google_storage_bucket_iam_policy.add-cf-storage-admin]
  zone_id = var.zoneid
  destination_conf = "gs://${replace(var.global_zonename,".","-")}-logs/logs"
}
data "google_storage_bucket_object_content" "challenge_data" {
  bucket = google_storage_bucket.log-storage.name
  name = cloudflare_logpush_ownership_challenge.ownership_challenge.ownership_challenge_filename

}
resource "cloudflare_logpush_job" "cf_logpush" {
  depends_on = [google_storage_bucket.log-storage]
  enabled = true
  zone_id = var.zoneid
  name = "all-logs"
  logpull_options = "fields=ParentRayID,RayID,SecurityLevel,SmartRouteColoID,ClientIP,EdgeStartTimestamp&timestamps=rfc3339,sample=1.0"
  destination_conf = "gs://${replace(var.global_zonename,".","-")}-logs/logs"
  ownership_challenge = data.google_storage_bucket_object_content.challenge_data.content
  dataset = "http_requests"
}

[bflad]

Hi folks 👋 There have been numerous changes and fixes to terraform-plugin-sdk and Terraform CLI since the versions mentioned in the original issue report. In particular, Terraform CLI 0.13.x handled data source references different than all other Terraform CLI versions. If you still believe there is an issue with the SDK, it would be great if you could verify it against Terraform CLI 1.1.x and terraform-plugin-sdk 2.11.0 so we can take a fresh look in a new issue (especially showing the terraform plan output of each reproduction step. Thanks!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.