SQL Database Instance fails with Not Found #167

bradgignac · 2017-06-29T13:40:07Z

I'm having some trouble getting the google_sql_database_instance resource to work. I believe I've traced it back to a poorly formed GET request to the operations endpoint, so I'm filing an issue here. I've included the full error report and debug logs below, but I believe this is the relevant snippet from the debug logs:

2017/06/29 08:26:06 [DEBUG] plugin: terraform: ---[ REQUEST ]---------------------------------------
2017/06/29 08:26:06 [DEBUG] plugin: terraform: GET http://www.googleapis.com/sql/v1beta4/projects//operations/16b6609c-b951-4432-a1d7-16de14328f60?alt=json HTTP/1.1
2017/06/29 08:26:06 [DEBUG] plugin: terraform: Host: www.googleapis.com
2017/06/29 08:26:06 [DEBUG] plugin: terraform: User-Agent: google-api-go-client/0.5 (darwin amd64) Terraform/0.9.9
2017/06/29 08:26:06 [DEBUG] plugin: terraform: Accept-Encoding: gzip
2017/06/29 08:26:06 [DEBUG] plugin: terraform: 
2017/06/29 08:26:06 [DEBUG] plugin: terraform: 
2017/06/29 08:26:06 [DEBUG] plugin: terraform: -----------------------------------------------------
2017/06/29 08:26:06 [DEBUG] plugin: terraform: google-provider (internal) 2017/06/29 08:26:06 [DEBUG] Google API Response Details:
2017/06/29 08:26:06 [DEBUG] plugin: terraform: ---[ RESPONSE ]--------------------------------------
2017/06/29 08:26:06 [DEBUG] plugin: terraform: HTTP/2.0 404 Not Found
2017/06/29 08:26:06 [DEBUG] plugin: terraform: Alt-Svc: quic=":443"; ma=2592000; v="38,37,36,35"
2017/06/29 08:26:06 [DEBUG] plugin: terraform: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
2017/06/29 08:26:06 [DEBUG] plugin: terraform: Content-Type: text/html; charset=UTF-8
2017/06/29 08:26:06 [DEBUG] plugin: terraform: Date: Thu, 29 Jun 2017 12:26:06 GMT
2017/06/29 08:26:06 [DEBUG] plugin: terraform: Expires: Mon, 01 Jan 1990 00:00:00 GMT
2017/06/29 08:26:06 [DEBUG] plugin: terraform: Pragma: no-cache
2017/06/29 08:26:06 [DEBUG] plugin: terraform: Server: GSE
2017/06/29 08:26:06 [DEBUG] plugin: terraform: Vary: Origin
2017/06/29 08:26:06 [DEBUG] plugin: terraform: Vary: X-Origin
2017/06/29 08:26:06 [DEBUG] plugin: terraform: X-Content-Type-Options: nosniff
2017/06/29 08:26:06 [DEBUG] plugin: terraform: X-Frame-Options: SAMEORIGIN
2017/06/29 08:26:06 [DEBUG] plugin: terraform: X-Xss-Protection: 1; mode=block
2017/06/29 08:26:06 [DEBUG] plugin: terraform: 
2017/06/29 08:26:06 [DEBUG] plugin: terraform: Not Found
2017/06/29 08:26:06 [DEBUG] plugin: terraform: -----------------------------------------------------

Notice that the request is being made to http://www.googleapis.com/sql/v1beta4/projects//operations/16b6609c-b951-4432-a1d7-16de14328f60?alt=json, which does not include the project segment of the URL, which is documented at https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/operations/get.

Terraform Version

▶ terraform -v                
Terraform v0.9.9

Affected Resource(s)

google_sql_database_instance

Terraform Configuration Files

/* Provider */

provider "gcp" {
  project = "rackspace-dev"
  region = "us-west1"
}

/* Cloud SQL */

resource "google_sql_database_instance" "master" {
  project = "rackspace-dev"
  region = "us-west1"

  settings {
    tier = "db-n1-standard-1"
  }
}

Debug Output

https://gist.github.com/bradgignac/b8e27c1339c19f846fe1842ea88f11c6

Expected Behavior

Terraform should wait for the database instance to complete provisioning, correctly update the statefile and report success.

Actual Behavior

Terraform is failing to read from the Operations API endpoint and immediately reports an error. The database instance spins up successfully, but the state file is never updated so Terraform doesn't know it exists.

Steps to Reproduce

terraform apply

Important Factoids

Credentials are being read from the GOOGLE_APPLICATION_CREDENTIALS environment variable.

References

@danawillow sent me hashicorp/terraform#13091 in GCP Community Slack. This appears to be a similar but not quite the same issue that was fixed in a previous release, so I'm not convinced I'm seeing the same thing. Also worth calling out that this fails every time.

The text was updated successfully, but these errors were encountered:

danawillow · 2017-06-30T17:30:04Z

Hey @bradgignac, I'm not able to reproduce this. Is it still happening for you? If so, can you also share the snippet of the initial POST request/response from the logs?

bradgignac · 2017-07-03T19:22:48Z

@danawillow I think this is the entire relevant snippet, including the initial POST:

2017/06/29 08:31:49 [DEBUG] apply: google_sql_database_instance.master: executing Apply
2017/06/29 08:31:49 [DEBUG] plugin: terraform: google-provider (internal) 2017/06/29 08:31:49 [DEBUG] No meta timeoutkey found in Apply()
2017/06/29 08:31:49 [DEBUG] plugin: terraform: google-provider (internal) 2017/06/29 08:31:49 [DEBUG] Google API Request Details:
2017/06/29 08:31:49 [DEBUG] plugin: terraform: ---[ REQUEST ]---------------------------------------
2017/06/29 08:31:49 [DEBUG] plugin: terraform: POST http://www.googleapis.com/sql/v1beta4/projects/rackspace-dev/instances?alt=json HTTP/1.1
2017/06/29 08:31:49 [DEBUG] plugin: terraform: Host: www.googleapis.com
2017/06/29 08:31:49 [DEBUG] plugin: terraform: User-Agent: google-api-go-client/0.5 (darwin amd64) Terraform/0.9.9
2017/06/29 08:31:49 [DEBUG] plugin: terraform: Content-Length: 162
2017/06/29 08:31:49 [DEBUG] plugin: terraform: Content-Type: application/json
2017/06/29 08:31:49 [DEBUG] plugin: terraform: Accept-Encoding: gzip
2017/06/29 08:31:49 [DEBUG] plugin: terraform: 
2017/06/29 08:31:49 [DEBUG] plugin: terraform: {"databaseVersion":"MYSQL_5_6","name":"terraform-00d7964078b83dbec0bc8157a2","region":"us-west1","settings":{"storageAutoResize":true,"tier":"db-n1-standard-1"}}
2017/06/29 08:31:49 [DEBUG] plugin: terraform: 
2017/06/29 08:31:49 [DEBUG] plugin: terraform: -----------------------------------------------------
2017/06/29 08:31:51 [DEBUG] plugin: terraform: google-provider (internal) 2017/06/29 08:31:51 [DEBUG] Google API Response Details:
2017/06/29 08:31:51 [DEBUG] plugin: terraform: ---[ RESPONSE ]--------------------------------------
2017/06/29 08:31:51 [DEBUG] plugin: terraform: HTTP/2.0 200 OK
2017/06/29 08:31:51 [DEBUG] plugin: terraform: Alt-Svc: quic=":443"; ma=2592000; v="39,38,37,36,35"
2017/06/29 08:31:51 [DEBUG] plugin: terraform: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
2017/06/29 08:31:51 [DEBUG] plugin: terraform: Content-Type: application/json; charset=UTF-8
2017/06/29 08:31:51 [DEBUG] plugin: terraform: Date: Thu, 29 Jun 2017 12:31:51 GMT
2017/06/29 08:31:51 [DEBUG] plugin: terraform: Etag: "7nzH-h2yIa30FGKFRs9YFu88s0g/NEsgZUf7G3oYgwYOf5j0TfAF2_4"
2017/06/29 08:31:51 [DEBUG] plugin: terraform: Expires: Mon, 01 Jan 1990 00:00:00 GMT
2017/06/29 08:31:51 [DEBUG] plugin: terraform: Pragma: no-cache
2017/06/29 08:31:51 [DEBUG] plugin: terraform: Server: GSE
2017/06/29 08:31:51 [DEBUG] plugin: terraform: Vary: Origin
2017/06/29 08:31:51 [DEBUG] plugin: terraform: Vary: X-Origin
2017/06/29 08:31:51 [DEBUG] plugin: terraform: X-Content-Type-Options: nosniff
2017/06/29 08:31:51 [DEBUG] plugin: terraform: X-Frame-Options: SAMEORIGIN
2017/06/29 08:31:51 [DEBUG] plugin: terraform: X-Xss-Protection: 1; mode=block
2017/06/29 08:31:51 [DEBUG] plugin: terraform: 
2017/06/29 08:31:51 [DEBUG] plugin: terraform: {
2017/06/29 08:31:51 [DEBUG] plugin: terraform:  "kind": "sql#operation",
2017/06/29 08:31:51 [DEBUG] plugin: terraform:  "selfLink": "https://www.googleapis.com/sql/v1beta4/projects/rackspace-dev/operations/9b17afd9-bdc9-428b-97ef-37133547c1a0",
2017/06/29 08:31:51 [DEBUG] plugin: terraform:  "targetProject": "rackspace-dev",
2017/06/29 08:31:51 [DEBUG] plugin: terraform:  "targetId": "terraform-00d7964078b83dbec0bc8157a2",
2017/06/29 08:31:51 [DEBUG] plugin: terraform:  "targetLink": "https://www.googleapis.com/sql/v1beta4/projects/rackspace-dev/instances/terraform-00d7964078b83dbec0bc8157a2",
2017/06/29 08:31:51 [DEBUG] plugin: terraform:  "name": "9b17afd9-bdc9-428b-97ef-37133547c1a0",
2017/06/29 08:31:51 [DEBUG] plugin: terraform:  "operationType": "CREATE",
2017/06/29 08:31:51 [DEBUG] plugin: terraform:  "status": "PENDING",
2017/06/29 08:31:51 [DEBUG] plugin: terraform:  "user": "[email protected]",
2017/06/29 08:31:51 [DEBUG] plugin: terraform:  "insertTime": "2017-06-29T12:31:51.630Z"
2017/06/29 08:31:51 [DEBUG] plugin: terraform: }
2017/06/29 08:31:51 [DEBUG] plugin: terraform: 
2017/06/29 08:31:51 [DEBUG] plugin: terraform: -----------------------------------------------------
2017/06/29 08:31:51 [DEBUG] plugin: terraform: google-provider (internal) 2017/06/29 08:31:51 [DEBUG] Waiting for state to become: [DONE]
2017/06/29 08:31:51 [DEBUG] plugin: terraform: google-provider (internal) 2017/06/29 08:31:51 [DEBUG] self_link: https://www.googleapis.com/sql/v1beta4/projects/rackspace-dev/operations/9b17afd9-bdc9-428b-97ef-37133547c1a0
2017/06/29 08:31:51 [DEBUG] plugin: terraform: google-provider (internal) 2017/06/29 08:31:51 [DEBUG] Google API Request Details:
2017/06/29 08:31:51 [DEBUG] plugin: terraform: ---[ REQUEST ]---------------------------------------
2017/06/29 08:31:51 [DEBUG] plugin: terraform: GET http://www.googleapis.com/sql/v1beta4/projects//operations/9b17afd9-bdc9-428b-97ef-37133547c1a0?alt=json HTTP/1.1
2017/06/29 08:31:51 [DEBUG] plugin: terraform: Host: www.googleapis.com
2017/06/29 08:31:51 [DEBUG] plugin: terraform: User-Agent: google-api-go-client/0.5 (darwin amd64) Terraform/0.9.9
2017/06/29 08:31:51 [DEBUG] plugin: terraform: Accept-Encoding: gzip
2017/06/29 08:31:51 [DEBUG] plugin: terraform: 
2017/06/29 08:31:51 [DEBUG] plugin: terraform: 
2017/06/29 08:31:51 [DEBUG] plugin: terraform: -----------------------------------------------------
2017/06/29 08:31:51 [DEBUG] plugin: terraform: google-provider (internal) 2017/06/29 08:31:51 [DEBUG] Google API Response Details:
2017/06/29 08:31:51 [DEBUG] plugin: terraform: ---[ RESPONSE ]--------------------------------------
2017/06/29 08:31:51 [DEBUG] plugin: terraform: HTTP/2.0 404 Not Found
2017/06/29 08:31:51 [DEBUG] plugin: terraform: Alt-Svc: quic=":443"; ma=2592000; v="39,38,37,36,35"
2017/06/29 08:31:51 [DEBUG] plugin: terraform: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
2017/06/29 08:31:51 [DEBUG] plugin: terraform: Content-Type: text/html; charset=UTF-8
2017/06/29 08:31:51 [DEBUG] plugin: terraform: Date: Thu, 29 Jun 2017 12:31:51 GMT
2017/06/29 08:31:51 [DEBUG] plugin: terraform: Expires: Mon, 01 Jan 1990 00:00:00 GMT
2017/06/29 08:31:51 [DEBUG] plugin: terraform: Pragma: no-cache
2017/06/29 08:31:51 [DEBUG] plugin: terraform: Server: GSE
2017/06/29 08:31:51 [DEBUG] plugin: terraform: Vary: Origin
2017/06/29 08:31:51 [DEBUG] plugin: terraform: Vary: X-Origin
2017/06/29 08:31:51 [DEBUG] plugin: terraform: X-Content-Type-Options: nosniff
2017/06/29 08:31:51 [DEBUG] plugin: terraform: X-Frame-Options: SAMEORIGIN
2017/06/29 08:31:51 [DEBUG] plugin: terraform: X-Xss-Protection: 1; mode=block
2017/06/29 08:31:51 [DEBUG] plugin: terraform: 
2017/06/29 08:31:51 [DEBUG] plugin: terraform: Not Found
2017/06/29 08:31:51 [DEBUG] plugin: terraform: -----------------------------------------------------

The full logs are available here at https://gist.github.com/bradgignac/b8e27c1339c19f846fe1842ea88f11c6.

bradgignac · 2017-07-03T19:38:42Z

@danawillow One other thing worth noting that might be another effect of whatever my problem is: every Cloud SQL resource complains if the project attribute isn't explicitly set. The docs indicate these are optional. Any possibility there's something about my provider setup that is causing issues specific to Cloud SQL resources?

danawillow · 2017-07-05T17:19:30Z

You always need a project set, but it doesn't have to be in the resource itself. Looks like you have it set in your provider config, so that should be enough:

provider "gcp" {
  project = "rackspace-dev"
  region = "us-west1"
}

If you remove it from the cloud sql resource, does it fail? Are you able to set up other resources successfully?

ubschmidt2 · 2017-07-14T07:04:30Z

FWIW, I've also encountered this issue. Haven't had a chance yet to look into more details.

joe-boyce · 2017-09-18T21:30:53Z

Same thing happening with the latest Terraform v0.10.5

Config:

Module

resource "google_sql_database_instance" "master" {
  project            = "${var.project_name}"
  name               = "${var.name}"
  database_version   = "MYSQL_5_7"
  region = "${var.region}"

  settings {
    tier             = "${var.instance_type}"

    backup_configuration {
        binary_log_enabled = "${var.binary_enabled}"
        enabled            = "${var.backup_enabled}"
        start_time         = "${var.backup_time}"
    }

  }
}

Use of module

module "cms_master" {
  source = "../../../../../global/modules/cloudsql_master"
  project_name = "*projectid*"
  name = "cms-master-instance"
  region = "us-central1"
  instance_type = "db-n1-standard-1"
  binary_enabled = "true"
  backup_enabled = "true"
  backup_time = "02:00"
}

Error when running as follows:

module.cms_master.google_sql_database_instance.master: Creating...
  database_version:                                     "" => "MYSQL_5_7"
  ip_address.#:                                         "" => "<computed>"
  master_instance_name:                                 "" => "<computed>"
  name:                                                 "" => "cms-master-instance"
  project:                                              "" => "*projectid*"
  region:                                               "" => "us-central1"
  replica_configuration.#:                              "" => "<computed>"
  self_link:                                            "" => "<computed>"
  settings.#:                                           "" => "1"
  settings.0.activation_policy:                         "" => "<computed>"
  settings.0.backup_configuration.#:                    "" => "1"
  settings.0.backup_configuration.0.binary_log_enabled: "" => "true"
  settings.0.backup_configuration.0.enabled:            "" => "true"
  settings.0.backup_configuration.0.start_time:         "" => "02:00"
  settings.0.crash_safe_replication:                    "" => "<computed>"
  settings.0.disk_autoresize:                           "" => "<computed>"
  settings.0.disk_size:                                 "" => "<computed>"
  settings.0.disk_type:                                 "" => "<computed>"
  settings.0.ip_configuration.#:                        "" => "<computed>"
  settings.0.location_preference.#:                     "" => "<computed>"
  settings.0.pricing_plan:                              "" => "PER_USE"
  settings.0.replication_type:                          "" => "SYNCHRONOUS"
  settings.0.tier:                                      "" => "db-n1-standard-1"
  settings.0.version:                                   "" => "<computed>"
Releasing state lock. This may take a few moments...
Error applying plan:

1 error(s) occurred:

* module.cms_master.google_sql_database_instance.master: 1 error(s) occurred:

* google_sql_database_instance.master: Error waiting for Create Instance (op 293cf935-a6b4-49ce-ac08-9f3f7f9327e7): googleapi: got HTTP response code 404 with body: Not Found

joe-boyce · 2017-09-18T23:16:20Z

This also happens when creating new databases & users:

* google_sql_database.database: Error, failure waiting for insertion of cms into cmsmaster: Error waiting for Insert Database (op 5057909a-1b93-4524-8fc6-845b09fad458): googleapi: got HTTP response code 404 with body: Not Found

* google_sql_user.users: Error, failure waiting for insertion of cms_dev into cmsmaster: Error waiting for Insert User (op ed9b1e4b-fea4-40e1-a0c3-c681e0eb806e): googleapi: got HTTP response code 404 with body: Not Found

But funnily enough those two resources do make it into the terraform state regardless of the errors above ^ and a subsequent plan shows no changes needed

joe-boyce · 2017-09-26T18:03:31Z

This one is still happening :(

Not sure if its in the track for a fix for the next provider release?

Cheers,

Joe

danawillow · 2017-09-26T18:22:39Z

No, not for the next release. I can try again to see if I can reproduce it but my efforts are better spent right now on issues that I can. If someone else wants to try debugging this please go for it.

joe-boyce · 2017-09-26T18:26:52Z

Let me know if you want any debug logs for this, trying with simple resource/module detailed in my comment above fails every time

joe-boyce · 2017-09-26T18:36:02Z

I believe that this is suffering from the same problem detailed in #436 whereby the project is missing as its in the config block rather than provider

Debug log shows the following:

2017-09-26T18:33:53.744Z [DEBUG] plugin.terraform-provider-google_v0.1.3_x4: 2017/09/26 18:33:53 [DEBUG] Google API Request Details:
2017-09-26T18:33:53.744Z [DEBUG] plugin.terraform-provider-google_v0.1.3_x4: ---[ REQUEST ]---------------------------------------
2017-09-26T18:33:53.744Z [DEBUG] plugin.terraform-provider-google_v0.1.3_x4: GET /sql/v1beta4/projects//operations/28b50cd8-ec1c-449c-853e-0ecc170edf8c?alt=json HTTP/1.1
2017-09-26T18:33:53.744Z [DEBUG] plugin.terraform-provider-google_v0.1.3_x4: Host: www.googleapis.com
2017-09-26T18:33:53.744Z [DEBUG] plugin.terraform-provider-google_v0.1.3_x4: User-Agent: google-api-go-client/0.5 (linux amd64) Terraform/0.10.0-dev
2017-09-26T18:33:53.744Z [DEBUG] plugin.terraform-provider-google_v0.1.3_x4: Accept-Encoding: gzip
2017-09-26T18:33:53.744Z [DEBUG] plugin.terraform-provider-google_v0.1.3_x4: 
2017-09-26T18:33:53.744Z [DEBUG] plugin.terraform-provider-google_v0.1.3_x4: 
2017-09-26T18:33:53.744Z [DEBUG] plugin.terraform-provider-google_v0.1.3_x4: -----------------------------------------------------
2017-09-26T18:33:53.822Z [DEBUG] plugin.terraform-provider-google_v0.1.3_x4: 2017/09/26 18:33:53 [DEBUG] Google API Response Details:
2017-09-26T18:33:53.822Z [DEBUG] plugin.terraform-provider-google_v0.1.3_x4: ---[ RESPONSE ]--------------------------------------
2017-09-26T18:33:53.822Z [DEBUG] plugin.terraform-provider-google_v0.1.3_x4: HTTP/2.0 404 Not Found

danawillow · 2017-09-26T19:24:01Z

Oh, it is. That was what I thought before but then I couldn't find where it was setting the project from the wrong place for some reason. Just found it now.

ghost · 2020-03-30T14:38:50Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!

danawillow self-assigned this Jun 30, 2017

ubschmidt2 mentioned this issue Jul 24, 2017

Project Name not set on Google Cloud SQL Database #233

Closed

danawillow mentioned this issue Sep 26, 2017

Fix a few instances where we read the project from the provider config and not using the helper function #469

Merged

danawillow closed this as completed in #469 Sep 27, 2017

HebertCL mentioned this issue Jul 25, 2019

Error messages from regular execution differ from error messages in debug mode #4108

Closed

ghost locked and limited conversation to collaborators Mar 30, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SQL Database Instance fails with Not Found #167

SQL Database Instance fails with Not Found #167

bradgignac commented Jun 29, 2017 •

edited

Loading

danawillow commented Jun 30, 2017

bradgignac commented Jul 3, 2017

bradgignac commented Jul 3, 2017

danawillow commented Jul 5, 2017

ubschmidt2 commented Jul 14, 2017

joe-boyce commented Sep 18, 2017 •

edited

Loading

joe-boyce commented Sep 18, 2017

joe-boyce commented Sep 26, 2017

danawillow commented Sep 26, 2017

joe-boyce commented Sep 26, 2017

joe-boyce commented Sep 26, 2017

danawillow commented Sep 26, 2017

ghost commented Mar 30, 2020

SQL Database Instance fails with Not Found #167

SQL Database Instance fails with Not Found #167

Comments

bradgignac commented Jun 29, 2017 • edited Loading

Terraform Version

Affected Resource(s)

Terraform Configuration Files

Debug Output

Expected Behavior

Actual Behavior

Steps to Reproduce

Important Factoids

References

danawillow commented Jun 30, 2017

bradgignac commented Jul 3, 2017

bradgignac commented Jul 3, 2017

danawillow commented Jul 5, 2017

ubschmidt2 commented Jul 14, 2017

joe-boyce commented Sep 18, 2017 • edited Loading

joe-boyce commented Sep 18, 2017

joe-boyce commented Sep 26, 2017

danawillow commented Sep 26, 2017

joe-boyce commented Sep 26, 2017

joe-boyce commented Sep 26, 2017

danawillow commented Sep 26, 2017

ghost commented Mar 30, 2020

bradgignac commented Jun 29, 2017 •

edited

Loading

joe-boyce commented Sep 18, 2017 •

edited

Loading