[Bug] elasticstack_elasticsearch_data_stream_lifecycle data_retention serialization issue on apply

[howtwizer]

Describe the bug
Truing to apply

resource "elasticstack_elasticsearch_data_stream_lifecycle" "proxy_log_data_stream_lifecycle_multiple" {
  name           = "logs-proxy-*"
  data_retention = "30d"
}

Failing with error

│ Error: json: cannot unmarshal object into Go value of type []models.DataStreamLifecycle
│ 
│   with elasticstack_elasticsearch_data_stream_lifecycle.proxy_log_data_stream_lifecycle_multiple,
│   on data-stream-lifecycles.tf line 1, in resource "elasticstack_elasticsearch_data_stream_lifecycle" "proxy_log_data_stream_lifecycle_multiple":
│    1: resource "elasticstack_elasticsearch_data_stream_lifecycle" "proxy_log_data_stream_lifecycle_multiple" {
│ 

To Reproduce
Steps to reproduce the behavior:

1. TF configuration used '

  required_version = ">= 0.12"
  required_providers {
    elasticstack = {
      source  = "elastic/elasticstack"
      version = "0.11.11"
    }
  }```'
2. TF operations to execute to get the error 'terraform apply' 
3. See the error in the output '

2024-11-22T13:02:01.692+0100 [ERROR] AttachSchemaTransformer: No provider config schema available for provider["terraform.io/builtin/terraform"]
2024-11-22T13:02:01.698+0100 [INFO] provider: configuring client automatic mTLS
2024-11-22T13:02:01.727+0100 [INFO] provider.terraform-provider-elasticstack_v0.11.11: configuring server automatic mTLS: timestamp="2024-11-22T13:02:01.726+0100"
elasticstack_elasticsearch_data_stream_lifecycle.proxy_log_data_stream_lifecycle_multiple: Creating...
2024-11-22T13:02:01.747+0100 [INFO] Starting apply for elasticstack_elasticsearch_data_stream_lifecycle.proxy_log_data_stream_lifecycle_multiple
2024-11-22T13:02:03.356+0100 [ERROR] provider.terraform-provider-elasticstack_v0.11.11: Response contains error diagnostic: @module=sdk.proto diagnostic_severity=ERROR diagnostic_summary="json: cannot unmarshal object into Go value of type []models.DataStreamLifecycle" tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/[email protected]/tfprotov6/internal/diag/diagnostics.go:58 diagnostic_detail="" tf_proto_version=6.6 tf_provider_addr=registry.terraform.io/elastic/elasticstack tf_req_id=e79a338f-0d81-e149-7716-e91b036fdd73 tf_resource_type=elasticstack_elasticsearch_data_stream_lifecycle timestamp="2024-11-22T13:02:03.356+0100"
2024-11-22T13:02:03.358+0100 [ERROR] vertex "elasticstack_elasticsearch_data_stream_lifecycle.proxy_log_data_stream_lifecycle_multiple" error: json: cannot unmarshal object into Go value of type []models.DataStreamLifecycle
╷
│ Error: json: cannot unmarshal object into Go value of type []models.DataStreamLifecycle
│
│ with elasticstack_elasticsearch_data_stream_lifecycle.proxy_log_data_stream_lifecycle_multiple,
│ on data-stream-lifecycles.tf line 1, in resource "elasticstack_elasticsearch_data_stream_lifecycle" "proxy_log_data_stream_lifecycle_multiple":
│ 1: resource "elasticstack_elasticsearch_data_stream_lifecycle" "proxy_log_data_stream_lifecycle_multiple" {
│
╵

**Expected behavior**
Create Data stream retention policy with 30d retention period for data streams "logs-proxy-*"

[tobio]

@howtwizer I'm struggling to reproduce this. Are you able to either:

• Include the full debug logs TF_LOG=debug terraform apply
• Include a full TF module definition and ES Stack version which reliably reproduces this error

Thanks for the report, the error suggests you're getting something different back from the API to what Elasticsearch specifies. The provider should definitely handle this better, but we need to understand the cause first.

[Tezno]

I have the same bug
@tobio I have followed your recommendation and I have run with TF_LOG=debug
As a result

• All call elastic http return 200
• Elastic retention on datastream are well set

Here's my error log

-----------------------------------------------------: @module=elasticstack tf_mux_provider="*proto6server.Server" tf_rpc=ApplyResourceChange @caller=github.com/elastic/terraform-provider-elasticstack/internal/clients/config/debug.go:70 tf_req_id=da1d5acf-c75a-541c-6079-d00a6782cd16 tf_resource_type=elasticstack_elasticsearch_data_stream_lifecycle tf_provider_addr=registry.terraform.io/elastic/elasticstack timestamp="2024-12-11T17:34:29.224+0100"
2024-12-11T17:34:29.248+0100 [DEBUG] provider.terraform-provider-elasticstack_v0.11.11: elasticsearch request [GET https://***/_data_stream/****/_lifecycle?expand_wildcards=open] executed. Took 23.684ms. <nil>: tf_mux_provider="*proto6server.Server" tf_provider_addr=registry.terraform.io/elastic/elasticstack tf_req_id=da1d5acf-c75a-541c-6079-d00a6782cd16 tf_resource_type=elasticstack_elasticsearch_data_stream_lifecycle tf_rpc=ApplyResourceChange @caller=github.com/elastic/terraform-provider-elasticstack/internal/clients/config/debug.go:37 @module=elasticstack timestamp="2024-12-11T17:34:29.247+0100"
2024-12-11T17:34:29.248+0100 [DEBUG] provider.terraform-provider-elasticstack_v0.11.11: elasticsearch API Response for [GET https://***/_data_stream/****/_lifecycle?expand_wildcards=open] Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 200 OK
Connection: close
Alt-Svc: h3=":443";ma=60;
Content-Type: application/json
X-Elastic-Product: Elasticsearch

{
 "global_retention": {},
 "data_streams": [
  {
   "name": "****",
   "lifecycle": {
    "enabled": true,
    "data_retention": "365d",
    "effective_retention": "365d",
    "retention_determined_by": "data_stream_configuration"
   }
  }
 ]
}
-----------------------------------------------------: tf_mux_provider="*proto6server.Server" tf_provider_addr=registry.terraform.io/elastic/elasticstack tf_req_id=da1d5acf-c75a-541c-6079-d00a6782cd16 tf_resource_type=elasticstack_elasticsearch_data_stream_lifecycle tf_rpc=ApplyResourceChange @caller=github.com/elastic/terraform-provider-elasticstack/internal/clients/config/debug.go:70 @module=elasticstack timestamp="2024-12-11T17:34:29.248+0100"
2024-12-11T17:34:29.248+0100 [ERROR] provider.terraform-provider-elasticstack_v0.11.11: Response contains error diagnostic: @caller=github.com/hashicorp/[email protected]/tfprotov6/internal/diag/diagnostics.go:58 diagnostic_summary="json: cannot unmarshal object into Go value of type []models.DataStreamLifecycle" tf_proto_version=6.6 tf_provider_addr=registry.terraform.io/elastic/elasticstack tf_rpc=ApplyResourceChange tf_resource_type=elasticstack_elasticsearch_data_stream_lifecycle @module=sdk.proto diagnostic_detail="" diagnostic_severity=ERROR tf_req_id=da1d5acf-c75a-541c-6079-d00a6782cd16 timestamp="2024-12-11T17:34:29.248+0100"
2024-12-11T17:34:29.250+0100 [DEBUG] State storage *remote.State declined to persist a state snapshot
2024-12-11T17:34:29.250+0100 [ERROR] vertex "elasticstack_elasticsearch_data_stream_lifecycle.lifecycle[\"****\"]" error: json: cannot unmarshal object into Go value of type []models.DataStreamLifecycle
2024-12-11T17:34:29.251+0100 [DEBUG] states/remote: state read serial is: 444; serial is: 444
2024-12-11T17:34:29.251+0100 [DEBUG] states/remote: state read lineage is: 000acdd3-cecd-22ce-bf36-e5aab31a2d03; lineage is: 000acdd3-cecd-22ce-bf36-e5aab31a2d03
╷
│ Warning: Applied changes may be incomplete
│ 
│ The plan was created with the -target option in effect, so some changes requested in the configuration may have been ignored and the output values may not be fully updated. Run the following command to verify that no other changes are pending:
│     terraform plan
│ 
│ Note that the -target option is not suitable for routine use, and is provided only for exceptional situations such as recovering from errors or mistakes, or when Terraform specifically suggests to use it as part of an error message.
╵
╷
│ Error: json: cannot unmarshal object into Go value of type []models.DataStreamLifecycle
│ 
│   with elasticstack_elasticsearch_data_stream_lifecycle.lifecycle["****"],
│   on es_datastreams.tf line 10, in resource "elasticstack_elasticsearch_data_stream_lifecycle" "lifecycle":
│   10: resource "elasticstack_elasticsearch_data_stream_lifecycle" "lifecycle" {

So I have investigate code and I found this

• I think there is mismatch in code between result of GET https://***/_data_stream/****/_lifecycle?expand_wildcards=open and go struct DataStreamLifecycle

Object return by call http:

{
 "global_retention": {},
 "data_streams": [
  {
   "name": "****",
   "lifecycle": {
    "enabled": true,
    "data_retention": "365d",
    "effective_retention": "365d",
    "retention_determined_by": "data_stream_configuration"
   }
  }
 ]
}

Go struct

• Unmarshal
• Struct

type DataStreamLifecycle struct {
	Name      string            `json:"name"`
	Lifecycle LifecycleSettings `json:"lifecycle,omitempty"`
}

[biscout42]

Folks, on which elastic stack version are you hitting the bug?