Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crash error: Root resource was present, but now absent. #8128

Assignees
Labels
bug forward/review In review; remove label to forward service/healthcare

Comments

@chaopeng
Copy link

chaopeng commented Jan 6, 2021

I have see a when "Root resource was present, but now absent." error when upgrade the version of terraform-provider-google to 3.51.0

In this PR terraform-google-modules/terraform-google-healthcare#36

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
  • Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment.
  • If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Affected Resource(s)

I don't know yet. It seems related to google_healthcare_dataset

Terraform Configuration Files

https://github.com/terraform-google-modules/terraform-google-healthcare/blob/master/examples/simple_example/main.tf

Debug Output

Panic Output

https://github.com/terraform-google-modules/terraform-google-healthcare/pull/36/checks?check_run_id=1658019589

Error: Provider produced inconsistent result after apply
 
When applying changes to
module.example.module.healthcare.google_healthcare_dataset.dataset, provider
"registry.terraform.io/hashicorp/google" produced an unexpected new value:
Root resource was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own issue tracker.

Expected Behavior

No error

Actual Behavior

Steps to Reproduce

  1. clone the https://github.com/terraform-google-modules/terraform-google-healthcare
  2. create a GCP with pubsub and healthcare API enable
  3. terraform apply

Important Factoids

References

@ghost ghost added the bug label Jan 6, 2021
@venkykuberan venkykuberan self-assigned this Jan 7, 2021
@venkykuberan
Copy link
Contributor

@chaopeng The information you provided isn't adequate enough to understand the issue . We need to know what resource is failing and provide the debug log to help us identify where its filling?

Please also provide the specific resource config or plan output to repro the issue on our end

@yeweidaniel
Copy link

Hi, I am also running into this error. With the tf script being:

module "danielye_test4_healthcare_dataset" {
  source  = "terraform-google-modules/healthcare/google"
  version = "~> 1.2.0"
 
  name     = "danielye-test4-healthcare-dataset"
  project  = module.project.project_id
  location = "us-central1"
 
  fhir_stores = [
    {
      name    = "intermediate-fhir-store"
      version = "R4"
 
      enable_update_create          = true
      disable_referential_integrity = true
      notification_config = {
        pubsub_topic = "projects/danielye-test4-pipeline/topics/${module.danielye_test4_intermediate_store_topic.topic}"
      }
    },
    {
      name    = "final-fhir-store"
      version = "R4"
 
      enable_update_create          = true
      disable_referential_integrity = true
    },
  ]
}

I get the following error:

Error: Provider produced inconsistent result after apply

When applying changes to module.danielye_test4_healthcare_dataset.google_healthcare_dataset.dataset, provider "registry.terraform.io/hashicorp/google" produced an unexpected new value: Root resource was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own issue tracker.

@ghost ghost removed waiting-response labels Jan 18, 2021
@xingao267
Copy link
Contributor

xingao267 commented Jan 18, 2021

I was able to reproduce the same issue with the following configs. My Terraform version is 0.14.4. And I think the same issue occurs with Terraform 0.13 as well.

The project itself matters. If a dataset has been successfully created in a project before, then the issue seems do not occur. So to reproduce the issue, create a new project every time.

terraform {
  required_version = ">=0.14"
  required_providers {
    google      = "~> 3.0"
    google-beta = "~> 3.0"
  }
}

module "project" {
  source                  = "github.com/terraform-google-modules/terraform-google-project-factory?ref=c41ba360a6bc6800a30d284b8fa23eb3ef5a8d7f"
  name                    = "my-project"
  org_id                  = ""
  folder_id               = "xxxxxx"
  billing_account         = "xxxxxx"
  lien                    = true
  default_service_account = "keep"
  create_project_sa       = false
  activate_apis = [
    "compute.googleapis.com",
    "healthcare.googleapis.com",
  ]
}

module "healthcare_dataset" {
  source  = "terraform-google-modules/healthcare/google"
  version = "~> 1.2.0"

  name     = "healthcare-dataset"
  project  = module.project.project_id
  location = "us-central1"

  fhir_stores = [
    {
      name    = "final-fhir-store"
      version = "R4"

      enable_update_create          = true
      disable_referential_integrity = true
    },
  ]
}

@xingao267
Copy link
Contributor

I can also reproduce the error with just the google_healthcare_dataset resource. So it should not be an issue in the healthcare module but in the provider. And I forgot to mention that the version of provider I'm using is 3.52.0.

module "project" {
  source                  = "github.com/terraform-google-modules/terraform-google-project-factory?ref=c41ba360a6bc6800a30d284b8fa23eb3ef5a8d7f"
  name                    = "my-project"
  org_id                  = ""
  folder_id               = "xxx"
  billing_account         = "xxx"
  lien                    = true
  default_service_account = "keep"
  create_project_sa       = false
  activate_apis = [
    "compute.googleapis.com",
    "healthcare.googleapis.com",
  ]
}

resource "google_healthcare_dataset" "default" {
  name     = "healthcare-dataset"
  location = "us-central1"
  project  = module.project.project_id
}

@xingao267
Copy link
Contributor

My guess is such safety check was introduced in Terraform 0.12 as https://www.terraform.io/docs/extend/terraform-0.12-compatibility.html#inaccurate-plans but not enforced so this resource was fine with Terraform 0.12. However, it's likely since Terraform 0.13, such safety check is enforced and a fix needs to be implemented for the google_healthcare_dataset resource.

@xingao267
Copy link
Contributor

Logs when I set TF_LOG=DEBUG

2021-01-18T22:25:26.325-0500 [INFO]  plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:26 [DEBUG] Google API Request Details:
---[ REQUEST ]---------------------------------------
POST /v1/projects/xingao-15/locations/us-central1/datasets?alt=json&datasetId=healthcare-dataset HTTP/1.1
Host: healthcare.googleapis.com
User-Agent: Terraform/0.14.4 (+https://www.terraform.io) Terraform-Plugin-SDK/2.4.0 terraform-provider-google/dev
Content-Length: 30
Content-Type: application/json
Accept-Encoding: gzip

{
 "name": "healthcare-dataset"
}

-----------------------------------------------------: timestamp=2021-01-18T22:25:26.325-0500
2021-01-18T22:25:30.831-0500 [INFO]  plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:30 [DEBUG] Google API Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Connection: close
Transfer-Encoding: chunked
Alt-Svc: h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Date: Tue, 19 Jan 2021 03:25:30 GMT
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Google-Backends: [2002:a05:663c:1056:b029:120:df74:ce0a]:4001,/bns/jc/borg/jc/bns/cloud-healthcare/prod-us-central1.cloud-healthcare-esf/2.esf,acyyzk8:443
X-Google-Dos-Service-Trace: main:cloud-healthcare-prod-us-central1-api
X-Google-Esf-Cloud-Client-Params: backend_service_name: "healthcare.googleapis.com" backend_fully_qualified_method: "google.cloud.healthcare.v1.dataset.DatasetService.CreateDataset"
X-Google-Gfe-Handshake-Trace: GFE: acyyzk8:443,Mentat oracle: [2002:a05:7500:1087:b029:0:c83a:566a]:9801
X-Google-Gfe-Request-Trace: acyyzk8:443,/bns/jc/borg/jc/bns/cloud-healthcare/prod-us-central1.cloud-healthcare-esf/2.esf,acyyzk8:443
X-Google-Gfe-Response-Body-Transformations: chunked
X-Google-Gfe-Response-Code-Details-Trace: response_code_set_by_backend
X-Google-Gfe-Service-Trace: cloud-healthcare-prod-us-central1-api
X-Google-Netmon-Label: /bns/jc/borg/jc/bns/cloud-healthcare/prod-us-central1.cloud-healthcare-esf/2:esf
X-Google-Security-Signals: FRAMEWORK=HTTPSERVER2
X-Google-Service: cloud-healthcare-prod-us-central1-api
X-Google-Session-Info: CJbH2orbBRoEGBAoCzpsEhloZWFsdGhjYXJlLmdvb2dsZWFwaXMuY29tGIrIz7ieFiJINzY0MDg2MDUxODUwLTZxcjRwNmdwaTZobjUwNnB0OGVqdXE4M2RpMzQxaHVyLmFwcHMuZ29vZ2xldXNlcmNvbnRlbnQuY29tSg86DTEvLzA0YmlKNFAxU36yAQ9LTGVicmRIVHFfT2YzZ0U
X-Google-Shellfish-Status: CA0gBEBG
X-Xss-Protection: 0

75
{
-15/locations/us-central1/datasets/healthcare-dataset/operations/14286000718916616193"
}

0


-----------------------------------------------------: timestamp=2021-01-18T22:25:30.830-0500
2021-01-18T22:25:30.831-0500 [INFO]  plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:30 [DEBUG] Retry Transport: Stopping retries, last request was successful: timestamp=2021-01-18T22:25:30.831-0500
2021-01-18T22:25:30.831-0500 [INFO]  plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:30 [DEBUG] Retry Transport: Returning after 1 attempts: timestamp=2021-01-18T22:25:30.831-0500
2021-01-18T22:25:30.831-0500 [INFO]  plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:30 [DEBUG] Finished creating Dataset "projects/xingao-15/locations/us-central1/datasets/healthcare-dataset": map[string]interface {}{"name":"projects/xingao-15/locations/us-central1/datasets/healthcare-dataset/operations/14286000718916616193"}: timestamp=2021-01-18T22:25:30.831-0500
2021-01-18T22:25:30.831-0500 [INFO]  plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:30 [DEBUG] Waiting for state to become: [success]: timestamp=2021-01-18T22:25:30.831-0500
2021-01-18T22:25:30.831-0500 [INFO]  plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:30 [DEBUG] Retry Transport: starting RoundTrip retry loop: timestamp=2021-01-18T22:25:30.831-0500
2021-01-18T22:25:30.831-0500 [INFO]  plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:30 [DEBUG] Retry Transport: request attempt 0: timestamp=2021-01-18T22:25:30.831-0500
2021-01-18T22:25:30.832-0500 [INFO]  plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:30 [DEBUG] Google API Request Details:
---[ REQUEST ]---------------------------------------
GET /v1/projects/xingao-15/locations/us-central1/datasets/healthcare-dataset?alt=json HTTP/1.1
Host: healthcare.googleapis.com
User-Agent: Terraform/0.14.4 (+https://www.terraform.io) Terraform-Plugin-SDK/2.4.0 terraform-provider-google/dev
Content-Type: application/json
Accept-Encoding: gzip


-----------------------------------------------------: timestamp=2021-01-18T22:25:30.832-0500
2021-01-18T22:25:31.042-0500 [INFO]  plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:31 [DEBUG] Google API Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Transfer-Encoding: chunked
Alt-Svc: h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Date: Tue, 19 Jan 2021 03:25:31 GMT
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Google-Backends: [2002:a05:663c:1056:b029:120:df74:ce0a]:4001,/bns/jc/borg/jc/bns/cloud-healthcare/prod-us-central1.cloud-healthcare-esf/1.esf,acyyzk16:443
X-Google-Dos-Service-Trace: main:cloud-healthcare-prod-us-central1-api
X-Google-Esf-Cloud-Client-Params: backend_service_name: "healthcare.googleapis.com" backend_fully_qualified_method: "google.cloud.healthcare.v1.dataset.DatasetService.GetDataset" is_dapper_traced: true
X-Google-Gfe-Handshake-Trace: GFE: acyyzk16:443,Mentat oracle: [2002:a05:7500:1109:b029:1:5f0c:f0d3]:9801
X-Google-Gfe-Request-Trace: acyyzk16:443,/bns/jc/borg/jc/bns/cloud-healthcare/prod-us-central1.cloud-healthcare-esf/1.esf,acyyzk16:443
X-Google-Gfe-Response-Body-Transformations: chunked
X-Google-Gfe-Response-Code-Details-Trace: response_code_set_by_backend
X-Google-Gfe-Service-Trace: cloud-healthcare-prod-us-central1-api
X-Google-Netmon-Label: /bns/jc/borg/jc/bns/cloud-healthcare/prod-us-central1.cloud-healthcare-esf/1:esf
X-Google-Security-Signals: FRAMEWORK=HTTPSERVER2
X-Google-Service: cloud-healthcare-prod-us-central1-api
X-Google-Session-Info: CJbH2orbBRoEGBAoCzpsEhloZWFsdGhjYXJlLmdvb2dsZWFwaXMuY29tGIrIz7ieFiJINzY0MDg2MDUxODUwLTZxcjRwNmdwaTZobjUwNnB0OGVqdXE4M2RpMzQxaHVyLmFwcHMuZ29vZ2xldXNlcmNvbnRlbnQuY29tSg86DTEvLzA0YmlKNFAxU36yAQ9LTGVicmRIVHFfT2YzZ0U
X-Google-Shellfish-Status: CA0gBEBG
race.corp.google.com/trace?host=jcts15-20020a306acf0000b02901292ad7646a&trace_id=0x12a8c88f3e4636cf&id=0xff8fd67c4397534b&start=2021-01-18_19:25:30&rpc_duration=0.041417
X-Xss-Protection: 0

1bc
{
  "error": {
    "code": 404,
    "message": "dataset not initialized",
    "status": "NOT_FOUND",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.DebugInfo",
        "detail": "[ORIGINAL ERROR] generic::not_found: dataset not initialized, cause (internal only): generic::not_found: dataset not initialized [google.rpc.error_details_ext] { message: \"dataset not initialized\" } 318741215: \"\""
      }
    ]
  }
}

0


-----------------------------------------------------: timestamp=2021-01-18T22:25:31.042-0500
2021-01-18T22:25:31.042-0500 [INFO]  plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:31 [DEBUG] Retry Transport: Stopping retries, last request failed with non-retryable error: googleapi: got HTTP response code 404 with body: HTTP/1.1 404 Not Found
Connection: close
Transfer-Encoding: chunked
Alt-Svc: h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Date: Tue, 19 Jan 2021 03:25:31 GMT
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Google-Backends: [2002:a05:663c:1056:b029:120:df74:ce0a]:4001,/bns/jc/borg/jc/bns/cloud-healthcare/prod-us-central1.cloud-healthcare-esf/1.esf,acyyzk16:443
X-Google-Dos-Service-Trace: main:cloud-healthcare-prod-us-central1-api
X-Google-Esf-Cloud-Client-Params: backend_service_name: "healthcare.googleapis.com" backend_fully_qualified_method: "google.cloud.healthcare.v1.dataset.DatasetService.GetDataset" is_dapper_traced: true
X-Google-Gfe-Handshake-Trace: GFE: acyyzk16:443,Mentat oracle: [2002:a05:7500:1109:b029:1:5f0c:f0d3]:9801
X-Google-Gfe-Request-Trace: acyyzk16:443,/bns/jc/borg/jc/bns/cloud-healthcare/prod-us-central1.cloud-healthcare-esf/1.esf,acyyzk16:443
X-Google-Gfe-Response-Body-Transformations: chunked
X-Google-Gfe-Response-Code-Details-Trace: response_code_set_by_backend
X-Google-Gfe-Service-Trace: cloud-healthcare-prod-us-central1-api
X-Google-Netmon-Label: /bns/jc/borg/jc/bns/cloud-healthcare/prod-us-central1.cloud-healthcare-esf/1:esf
X-Google-Security-Signals: FRAMEWORK=HTTPSERVER2
X-Google-Service: cloud-healthcare-prod-us-central1-api
X-Google-Session-Info: CJbH2orbBRoEGBAoCzpsEhloZWFsdGhjYXJlLmdvb2dsZWFwaXMuY29tGIrIz7ieFiJINzY0MDg2MDUxODUwLTZxcjRwNmdwaTZobjUwNnB0OGVqdXE4M2RpMzQxaHVyLmFwcHMuZ29vZ2xldXNlcmNvbnRlbnQuY29tSg86DTEvLzA0YmlKNFAxU36yAQ9LTGVicmRIVHFfT2YzZ0U
fish-Status: CA0gBEBG
X-Google-Trace: http://trace.corp.google.com/trace?host=jcts15-20020a306acf0000b02901292ad7646a&trace_id=0x12a8c88f3e4636cf&id=0xff8fd67c4397534b&start=2021-01-18_19:25:30&rpc_duration=0.041417
X-Xss-Protection: 0

1bc
{
  "error": {
    "code": 404,
    "message": "dataset not initialized",
    "status": "NOT_FOUND",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.DebugInfo",
        "detail": "[ORIGINAL ERROR] generic::not_found: dataset not initialized, cause (internal only): generic::not_found: dataset not initialized [google.rpc.error_details_ext] { message: \"dataset not initialized\" } 318741215: \"\""
      }
    ]
  }
}

0
: timestamp=2021-01-18T22:25:31.042-0500
2021-01-18T22:25:31.042-0500 [INFO]  plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:31 [DEBUG] Retry Transport: Returning after 1 attempts: timestamp=2021-01-18T22:25:31.042-0500
2021-01-18T22:25:31.043-0500 [INFO]  plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:31 [WARN] Removing HealthcareDataset "projects/xingao-15/locations/us-central1/datasets/healthcare-dataset" because it's gone: timestamp=2021-01-18T22:25:31.042-0500
2021/01/18 22:25:31 [DEBUG] google_healthcare_dataset.default: apply errored, but we're indicating that via the Error pointer rather than returning it: Provider produced inconsistent result after apply: When applying changes to google_healthcare_dataset.default, provider "registry.terraform.io/hashicorp/google" produced an unexpected new value: Root resource was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own issue tracker.

Error: Provider produced inconsistent result after apply

When applying changes to google_healthcare_dataset.default, provider
"registry.terraform.io/hashicorp/google" produced an unexpected new value:
Root resource was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

2021-01-18T22:25:31.056-0500 [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-01-18T22:25:31.059-0500 [DEBUG] plugin: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/google/3.52.0/linux_amd64/terraform-provider-google_v3.52.0_x5 pid=2149476
2021-01-18T22:25:31.059-0500 [DEBUG] plugin: plugin exited

@xingao267
Copy link
Contributor

xingao267 commented Jan 19, 2021

The dataset was actually created successfully, but Terraform's GET call to verify it's created failed due to dataset not initialized . I guess some sort of retry logic should be implemented to retry the GET request so it doesn't fail and just exit in the first attempt.

@xingao267
Copy link
Contributor

@venkykuberan @rileykarson have you seen similar issue before? I can take a stab in fixing this, but might need some guidance from you or existing examples for how to add retry logics for a resource in magic-module.

@xingao267
Copy link
Contributor

xingao267 commented Jan 19, 2021

@xingao267
Copy link
Contributor

Worked. Sent GoogleCloudPlatform/magic-modules#4412

@xingao267
Copy link
Contributor

The fix is planned to be released in google provider 3.54.0 next Monday.

@ghost
Copy link

ghost commented Feb 20, 2021

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!

@ghost ghost locked as resolved and limited conversation to collaborators Feb 20, 2021
@github-actions github-actions bot added forward/review In review; remove label to forward service/healthcare labels Jan 14, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.