Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] elasticstack_elasticsearch_logstash_pipeline "Failed to decode resource from state" #334

Closed
gaima8 opened this issue May 25, 2023 · 2 comments
Labels
bug Something isn't working

Comments

@gaima8
Copy link

gaima8 commented May 25, 2023

Describe the bug
Upgrading to v0.6.0, from v0.5.0, to resolve #266 causes terraform to fail with a new (and exciting?) error.

╷
│ Error: Failed to decode resource from state
│ 
│ Error decoding
│ "module.elasticstack-prod.elasticstack_elasticsearch_logstash_pipeline.bob"
│ from previous state: string is required
╵
╷
│ Error: string is required
│ 
│   with module.elasticstack-prod.elasticstack_elasticsearch_logstash_pipeline.bob,
│   on modules/elasticstack-prod/logstash_pipelines.tf line 1, in resource "elasticstack_elasticsearch_logstash_pipeline" "bob":
│    1: resource "elasticstack_elasticsearch_logstash_pipeline" "bob" {
│ 
╵

Outside of the version of the provider the entirety of the change is;

@@ -10,8 +10,7 @@ resource "elasticstack_elasticsearch_logstash_pipeline" "bob" {
   pipeline_ecs_compatibility = "v1"
   pipeline_workers           = 24
   queue_drain                = true
-  queue_max_bytes_number     = 32
-  queue_max_bytes_units      = "gb"
+  queue_max_bytes            = "32gb"
   queue_max_events           = 0
   queue_page_capacity        = "124mb"
   queue_type                 = "persisted"

Debug output
With TF_LOG=trace the output is over 10000 lines so I'm not posting that. I did find this in it however.

2023-05-25T10:23:55.3480708Z 2023-05-25T10:23:55.172Z [TRACE] GRPCProvider: ReadResource
2023-05-25T10:23:55.3481563Z 2023-05-25T10:23:55.172Z [ERROR] vertex "module.elasticstack-prod.elasticstack_elasticsearch_logstash_pipeline.bob" error: string is required
2023-05-25T10:23:55.3482329Z 2023-05-25T10:23:55.172Z [TRACE] vertex "module.elasticstack-prod.elasticstack_elasticsearch_logstash_pipeline.bob": visit complete, with errors
2023-05-25T10:23:55.3482851Z 2023-05-25T10:23:55.172Z [TRACE] dag/walk: upstream of "root" errored, so skipping
2023-05-25T10:23:55.3483745Z 2023-05-25T10:23:55.172Z [TRACE] vertex "module.elasticstack-prod.elasticstack_elasticsearch_logstash_pipeline.bob (expand)": dynamic subgraph encountered errors: string is required
2023-05-25T10:23:55.3484520Z 2023-05-25T10:23:55.172Z [ERROR] vertex "module.elasticstack-prod.elasticstack_elasticsearch_logstash_pipeline.bob (expand)" error: string is required
2023-05-25T10:23:55.3485327Z 2023-05-25T10:23:55.172Z [TRACE] vertex "module.elasticstack-prod.elasticstack_elasticsearch_logstash_pipeline.bob (expand)": visit complete, with errors

Versions (please complete the following information):

  • OS: [Github Actions]
  • Terraform Version [1.4.2]
  • Provider version [0.6.0]
  • Elasticsearch Version [8.6.2]

Additional context
terraform state show module.elasticstack-prod.elasticstack_elasticsearch_logstash_pipeline.bob also returns

string is required
# module.elasticstack-prod.elasticstack_elasticsearch_logstash_pipeline.bob:
resource "elasticstack_elasticsearch_logstash_pipeline" "bob" {
@gaima8 gaima8 added the bug Something isn't working label May 25, 2023
@RobsonSutton
Copy link
Contributor

Hey @gaima8 👋

As a part of the recent version bump, I introduced a breaking change to fix an issue where the pipeline metadata field would throw an error when manual changes were made to pipelines (#278).

In order to resolve this issue, the data type of the field has been changed. Unfortunately during my testing, I missed the test case for existing resources in state migrating from a previous version (where this field would be defined as TypeMap) to the new version (where this field is expected to be defined as a TypeString) - although to be honest I'm not sure if this could have been worked around anyway! It seems Terraform gets its head scrambled a bit due to this mismatch (when running a plan with existing resources created using an older provider version).

The workaround to this error would be to:

THEN

  • Run terraform plan (preferred)
  • Run terraform apply (preferred)

OR

  • Run terraform import as defined in the doc's for each pipeline

This will remove the reference to the resource in state (so won't impact the existing pipelines in elastic), the subsequent apply will then re-apply the resource with the new TypeString field (because the resources ID's will match in the cluster and the provider is leveraging a PUT request, there shouldn't be a need to run terraform import but it's another option available to then start managing these pipelines with Terraform again).

@tobio
Copy link
Member

tobio commented May 30, 2023

These instructions have been bundled into the release notes for 0.6.1

@tobio tobio closed this as completed May 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants