Add GCS ingestion settings and platform log settings to google_pubsub_topic

[andreigurau]

Add GCS ingestion settings and Platform Log Settings to google_pubsub_topic

Release Note Template for Downstream PRs (will be copied)

pubsub: added GCS ingestion settings and platform log settings to `google_pubsub_topic` resource

[github-actions]

Hello! I am a robot. Tests will require approval from a repository maintainer to run.

@NickElliot, a repository maintainer, has been assigned to review your changes. If you have not received review feedback within 2 business days, please leave a comment on this PR asking them to take a look.

You can help make sure that review is quick by doing a self-review and by running impacted tests locally.

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 4 files changed, 557 insertions(+))
google-beta provider: Diff ( 4 files changed, 557 insertions(+))
terraform-google-conversion: Diff ( 1 file changed, 156 insertions(+))
Open in Cloud Shell: Diff ( 4 files changed, 119 insertions(+))

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 4 files changed, 557 insertions(+))
google-beta provider: Diff ( 4 files changed, 557 insertions(+))
terraform-google-conversion: Diff ( 1 file changed, 156 insertions(+))
Open in Cloud Shell: Diff ( 4 files changed, 119 insertions(+))

[modular-magician]

Tests analytics

Total tests: 54
Passed tests: 51
Skipped tests: 1
Affected tests: 2

Click here to see the affected service packages

• pubsub

Action taken

Found 2 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

• TestAccPubsubTopic_cloudStorageIngestionUpdate
• TestAccPubsubTopic_pubsubTopicIngestionCloudStorageExample

Get to know how VCR tests work

[modular-magician]

Tests analytics

Total tests: 54
Passed tests: 51
Skipped tests: 1
Affected tests: 2

Click here to see the affected service packages

• pubsub

Action taken

Found 2 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

• TestAccPubsubTopic_cloudStorageIngestionUpdate
• TestAccPubsubTopic_pubsubTopicIngestionCloudStorageExample

Get to know how VCR tests work

[modular-magician]

$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccPubsubTopic_cloudStorageIngestionUpdate[Error message] [Debug log]
TestAccPubsubTopic_pubsubTopicIngestionCloudStorageExample[Error message] [Debug log]

$\textcolor{red}{\textsf{Errors occurred during RECORDING mode. Please fix them to complete your PR.}}$

View the build log or the debug log for each test

[modular-magician]

$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccPubsubTopic_cloudStorageIngestionUpdate[Error message] [Debug log]
TestAccPubsubTopic_pubsubTopicIngestionCloudStorageExample[Error message] [Debug log]

$\textcolor{red}{\textsf{Errors occurred during RECORDING mode. Please fix them to complete your PR.}}$

View the build log or the debug log for each test

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 4 files changed, 560 insertions(+))
google-beta provider: Diff ( 4 files changed, 560 insertions(+))
terraform-google-conversion: Diff ( 1 file changed, 156 insertions(+))
Open in Cloud Shell: Diff ( 4 files changed, 119 insertions(+))

[modular-magician]

Tests analytics

Total tests: 54
Passed tests: 51
Skipped tests: 1
Affected tests: 2

Click here to see the affected service packages

• pubsub

Action taken

Found 2 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

• TestAccPubsubTopic_cloudStorageIngestionUpdate
• TestAccPubsubTopic_pubsubTopicIngestionCloudStorageExample

Get to know how VCR tests work

[modular-magician]

$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccPubsubTopic_cloudStorageIngestionUpdate[Error message] [Debug log]
TestAccPubsubTopic_pubsubTopicIngestionCloudStorageExample[Error message] [Debug log]

$\textcolor{red}{\textsf{Errors occurred during RECORDING mode. Please fix them to complete your PR.}}$

View the build log or the debug log for each test

[andreigurau]

I'm not sure why we're getting 500s in the VCR-test. I tried to repro, but I can't get a 500 to occur, even when making the request using the same parameters copied over (the only difference being the project and bucket) (tested on the GCP API docs. I get 200s this way.

The tests are working for me locally, if I set the PROJECT_NAME env variable to a project with access, and set the bucket name to a valid bucket within PROJECT_NAME

[c2thorn]

I'm not sure why we're getting 500s in the VCR-test. I tried to repro, but I can't get a 500 to occur, even when making the request using the same parameters copied over (the only difference being the project and bucket) (tested on the GCP API docs. I get 200s this way.

The tests are working for me locally, if I set the PROJECT_NAME env variable to a project with access, and set the bucket name to a valid bucket within PROJECT_NAME

(I'm copy-pasting an announcement on each open pull request, but wanted to respond to this first)

Have you tried generating the provider with your change and testing it with your project? That would help determine if it is a Terraform implementation issue or an environment issue

(Going to paste my announcement below)

[c2thorn]

Tomorrow, the Magic Modules repository is scheduled to undergo a language migration from Ruby to Go. You can view more details about this in our announcement here: hashicorp/terraform-provider-google#19583 (or go/mm-migration-announcement if you are a Googler)

This open pull request may become incompatible due to most YAML and .erb files converting to Go-compatible files.

Our team (Magic Modules repository maintainers) has tooling to automatically convert changes to the new language, and we can prepare a new commit for this pull request that is compatible with the migration.

In order to push the new changes to your pull request, we would need to force push the commit to your fork's branch. Our tooling saves a backup branch before converting, so we could rollback or open a new pull request if needed. We would also work with you and the PR reviewer in the event additional changes are needed.

You also have the option to update the pull request yourself after the migration. You can view a preview branch and updated documentation related to the migration changes.

We will take no action until we have your explicit permission to push changes to your fork's branch used for this pull request. Let me or your reviewer know if you have any further questions!

[andreigurau]

Have you tried generating the provider with your change and testing it with your project? That would help determine if it is a Terraform implementation issue or an environment issue

Yeah, that's what I meant. After generating the provider, TestAccPubsubTopic_pubsubTopicIngestionCloudStorageExample and TestAccPubsubTopic_cloudStorageIngestionUpdate are both working for me locally, provided I set the bucket name + project name to valid values.

The one other update I have is that someone debugged the issue internally, and requests are being made (from the VCR-tests) that indicate that all the fields inside of ingestion_data_source_settings are being dropped when the call is made to our API

edit: I could be convinced to have the changes pushed here in order to test if the Go migration will fix the issue, but if there is a more clear solution, I would prefer to implement that first (edit again: To clarify, what I'm saying is don't push the change unless we want to check if the change will fix the issue)

[c2thorn]

Yeah, that's what I meant. After generating the provider, TestAccPubsubTopic_pubsubTopicIngestionCloudStorageExample and TestAccPubsubTopic_cloudStorageIngestionUpdate are both working for me locally, provided I set the bucket name + project name to valid values.

The one other update I have is that someone debugged the issue internally, and requests are being made (from the VCR-tests) that indicate that all the fields inside of ingestion_data_source_settings are being dropped when the call is made to our API

In one of the requests that return a 500, I'm seeing the request body as

PUT /v1/projects/<project-name>/topics/tf-test-topic-4ppkyqjskg?alt=json HTTP/1.1

{
 "ingestionDataSourceSettings": {
  "cloudStorage": {
   "avroFormat": null,
   "bucket": "test-bucket",
   "matchGlob": "foo/**",
   "minimumObjectCreateTime": "2024-01-01T00:00:00Z",
   "pubsubAvroFormat": null,
   "textFormat": {
    "delimiter": " "
   }
  },
  "platformLogsSettings": {
   "severity": "WARNING"
  }
 },
 "labels": {
  "goog-terraform-provisioned": "true"
 }
}

Is the body incorrect? Maybe the avroFormat fields?

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 4 files changed, 562 insertions(+))
google-beta provider: Diff ( 4 files changed, 562 insertions(+))
terraform-google-conversion: Diff ( 1 file changed, 156 insertions(+))
Open in Cloud Shell: Diff ( 4 files changed, 119 insertions(+))

[modular-magician]

Tests analytics

Total tests: 54
Passed tests: 51
Skipped tests: 1
Affected tests: 2

Click here to see the affected service packages

• pubsub

Action taken

Found 2 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

• TestAccPubsubTopic_cloudStorageIngestionUpdate
• TestAccPubsubTopic_pubsubTopicIngestionCloudStorageExample

Get to know how VCR tests work

[modular-magician]

$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccPubsubTopic_cloudStorageIngestionUpdate[Error message] [Debug log]
TestAccPubsubTopic_pubsubTopicIngestionCloudStorageExample[Error message] [Debug log]

$\textcolor{red}{\textsf{Errors occurred during RECORDING mode. Please fix them to complete your PR.}}$

View the build log or the debug log for each test

[andreigurau]

I don't think the body is malformed in that it throws 500s. When I run the TestAccPubsubTopic_cloudStorageIngestionUpdate test locally, this is the output I see when creating the Pubsub topic (results for TestAccPubsubTopic_pubsubTopicIngestionCloudStorageExample are similar to this as well):

---[ REQUEST ]---------------------------------------
PUT /v1/projects/<redacted project>/topics/tf-test-topic-c70fnwxfck?alt=json HTTP/1.1
Host: pubsub.googleapis.com
User-Agent: Terraform/1.9.6 (+https://www.terraform.io) Terraform-Plugin-SDK/2.33.0 terraform-provider-google-beta/acc
Content-Length: 315
Content-Type: application/json
Accept-Encoding: gzip

{
 "ingestionDataSourceSettings": {
  "cloudStorage": {
   "avroFormat": null,
   "bucket": "<redacted bucket>",
   "minimumObjectCreateTime": "2024-01-01T00:00:00Z",
   "pubsubAvroFormat": null,
   "textFormat": {
    "delimiter": " "
   }
  },
  "platformLogsSettings": {
   "severity": "WARNING"
  }
 },
 "labels": {
  "goog-terraform-provisioned": "true"
 }
}

-----------------------------------------------------
2024/09/24 22:17:35 [DEBUG] Google API Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 200 OK
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Date: Wed, 25 Sep 2024 02:17:35 GMT
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

{
  "name": "projects/<redacted project>/topics/tf-test-topic-c70fnwxfck",
  "labels": {
    "goog-terraform-provisioned": "true"
  },
  "ingestionDataSourceSettings": {
    "cloudStorage": {
      "bucket": "<redacted bucket>",
      "textFormat": {
        "delimiter": " "
      },
      "minimumObjectCreateTime": "2024-01-01T00:00:00Z"
    },
    "platformLogsSettings": {
      "severity": "WARNING"
    }
  }
}

-----------------------------------------------------
2024/09/24 22:17:35 [DEBUG] Retry Transport: Stopping retries, last request was successful
2024/09/24 22:17:35 [DEBUG] Retry Transport: Returning after 1 attempts
2024/09/24 22:17:35 [DEBUG] Creating Topic: Polling until expected state is read
2024/09/24 22:17:35 [DEBUG] Target occurrences: 1
2024/09/24 22:17:35 [DEBUG] Waiting for state to become: [success]
2024/09/24 22:17:35 [DEBUG] Waiting for state to become: [success]
2024/09/24 22:17:35 [DEBUG] Retry Transport: starting RoundTrip retry loop
2024/09/24 22:17:35 [DEBUG] Retry Transport: request attempt 0
2024/09/24 22:17:35 [DEBUG] Google API Request Details:
---[ REQUEST ]---------------------------------------
GET /v1/projects/<redacted project>/topics/tf-test-topic-c70fnwxfck?alt=json HTTP/1.1
Host: pubsub.googleapis.com
User-Agent: Terraform/1.9.6 (+https://www.terraform.io) Terraform-Plugin-SDK/2.33.0 terraform-provider-google-beta/acc
Content-Type: application/json
Accept-Encoding: gzip


-----------------------------------------------------
2024/09/24 22:17:35 [DEBUG] Google API Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 200 OK
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Date: Wed, 25 Sep 2024 02:17:35 GMT
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

{
  "name": "projects/<redacted project>/topics/tf-test-topic-c70fnwxfck",
  "labels": {
    "goog-terraform-provisioned": "true"
  },
  "ingestionDataSourceSettings": {
    "cloudStorage": {
      "bucket": "<redacted bucket>",
      "textFormat": {
        "delimiter": " "
      },
      "minimumObjectCreateTime": "2024-01-01T00:00:00Z"
    },
    "platformLogsSettings": {
      "severity": "WARNING"
    }
  }
}

This PR is based off of #9985, except that PR doesn't throw 500s. I'm still trying to debug internally as well, but if you have any other suggestions on how to debug here, let me know!

[andreigurau]

Turns out that internally we are rolling out a change to production that should cause these fields to be settable publicly, which I believe should be complete by end of week. We can try testing again after the rollout has finished.

I'm fine with you pushing out the Go migration change in the meantime

[NickElliot]

backup located here: https://github.com/NickElliot/magic-modules/tree/gcs-ingestion-backup
hiccup in order of operations for committing and pushing the changes, attempting to resolve now, everything is preserved have no fear

[c2thorn]

Hi @andreigurau, we've lost access to the branch. Would you mind adding us as collaborators at https://github.com/andreigurau/magic-modules/settings/access

[c2thorn]

Please add @NickElliot and @c2thorn
Then we should be able to restore the branch

[c2thorn]

The branch is converted, but this specific pull request is permanently closed. https://github.com/andreigurau/magic-modules/tree/gcs-ingestion is ready and converted for another pull request.