ec_deployment: Add autoscaling support

docs/data-sources/ec_deployment.md

@@ -38,6 +38,7 @@ These will not be available for interpolation.
   * `observability.#.logs` - Defines whether logs are enabled or disabled.
   * `observability.#.metrics` - Defines whether metrics are enabled or disabled.
 * `elasticsearch` - Instance configuration of the Elasticsearch resource kind.
+  * `elasticsearch.#.autoscale` - Whether or not Elasticsearch autoscaling is enabled.
   * `elasticsearch.#.healthy` - Resource kind health status.
   * `elasticsearch.#.cloud_id` - The encoded Elasticsearch credentials to use in Beats or Logstash. See [Configure Beats and Logstash with Cloud ID](https://www.elastic.co/guide/en/cloud/current/ec-cloud-id.html) for more information.
   * `elasticsearch.#.http_endpoint` - HTTP endpoint for the resource kind.
@@ -55,6 +56,11 @@ These will not be available for interpolation.
     * `elasticsearch.#.topology.#.node_type_master` - Defines whether this node can be elected master (<7.10.0).
     * `elasticsearch.#.topology.#.node_type_ingest` - Defines whether this node can run an ingest pipeline (<7.10.0).
     * `elasticsearch.#.topology.#.node_type_ml` - Defines whether this node can run ML jobs (<7.10.0).
+    * `elasticsearch.#.topology.#.autoscaling.#.max_size` - The maximum size for the scale up policy.
+    * `elasticsearch.#.topology.#.autoscaling.#.max_size_resource` - The maximum size resource for the scale up policy.
+    * `elasticsearch.#.topology.#.autoscaling.#.min_size` - The minimum size for the scale down policy.
+    * `elasticsearch.#.topology.#.autoscaling.#.min_size_resource` - The minimum size for the scale down policy.
+    * `elasticsearch.#.topology.#.autoscaling.#.policy_override_json` - The advanced policy overrides for the autoscaling policy.
 * `kibana` - Instance configuration of the Kibana type.
   * `kibana.#.elasticsearch_cluster_ref_id` - The user-specified ID of the Elasticsearch cluster to which this resource kind will link.
   * `kibana.#.healthy` - Resource kind health status.

docs/resources/ec_deployment.md

@@ -45,7 +45,7 @@ resource "ec_deployment" "example_minimal" {
 }
 ```
 
-### Hot warm cold tiered deployment
+### Tiered deployment with Autoscaling enabled
 
 ```hcl
 data "ec_stack" "latest" {
@@ -59,19 +59,26 @@ resource "ec_deployment" "example_minimal" {
   deployment_template_id = "aws-io-optimized-v2"
 
   elasticsearch {
+    autoscale = "true"
+
     topology {
-      id   = "hot_content"
+      id   = "cold"
       size = "8g"
     }
 
     topology {
-      id   = "warm"
-      size = "16g"
+      id   = "hot_content"
+      size = "8g"
+
+      autoscaling {
+        // Optionally change the policy max size.
+        // max_size = "29g"
+      }
     }
 
     topology {
-      id   = "cold"
-      size = "8g"
+      id   = "warm"
+      size = "16g"
     }
   }
 
@@ -207,15 +214,15 @@ The following arguments are supported:
 
 ### Resources
 
-!> **Warning on removing explicit topology objects** Due to current limitations, if a topology object is removed from the configuration, the removal won't trigger any changes since the field is optional and computed. There is no way to determine if the block was removed, which results in a _"sticky"_ topology configuration.
+!> **Warning on removing explicit topology objects** Due to current limitations, if a topology object is removed from the configuration, the removal won't trigger any changes since the field is optional and computed. There is no way to determine if the block was removed, which results in a _"sticky"_ topology configuration. To disable a topology element, set the `topology.size` to `"0g"`.
 
 To create a valid deployment, you must specify at least the resource type `elasticsearch`. The supported resources are listed below.
 
 A default topology from the deployment template is used for empty blocks: `elasticsearch {}`, `kibana {}`, `apm {}`, `enterprise_search {}`. When a block is not set, the resource kind is not enabled in the deployment.
 
 The `ec_deployment` resource will opt-out all the resources except Elasticsearch, which inherits the default topology from the deployment template. For example, the [I/O Optimized template includes an Elasticsearch cluster 8 GB memory x 2 availability zones](https://www.elastic.co/guide/en/cloud/current/ec-getting-started-profiles.html#ec-getting-started-profiles-io).
 
-To customize the size or settings of the deployment resource, use the `topology` block within each resource kind block.
+To customize the size or settings of the deployment resource, use the `topology` block within each resource kind block. The `topology` blocks are ordered lists and should be defined in the Terraform configuration in an ascending manner by alphabetical order of the `id` field.
 
 #### Elasticsearch
 
@@ -227,6 +234,7 @@ The required `elasticsearch` block supports the following arguments:
 * `remote_cluster` (Optional) Elasticsearch remote clusters to configure for the Elasticsearch resource. Can be set multiple times.
 * `snapshot_source` (Optional) Restores data from a snapshot of another deployment.
 * `extension` (Optional) Custom Elasticsearch bundles or plugins. Can be set multiple times.
+* `autoscale` (Optional) Enable or disable autoscaling. Defaults to the setting coming from the deployment template. Accepted values are `"true"` or `"false"`.
 
 ##### Topology
 
@@ -244,9 +252,23 @@ The optional `elasticsearch.topology` block supports the following arguments:
 * `node_type_master` - (Optional) The node type for the Elasticsearch cluster (master node).
 * `node_type_ingest` - (Optional) The node type for the Elasticsearch cluster (ingest node).
 * `node_type_ml` - (Optional) The node type for the Elasticsearch cluster (machine learning node).
+* `autoscaling` - (Optional) Autoscaling policy defining the maximum and / or minimum total size for this topology element. For more information refer to the `autoscaling` block.
 
 ~> **Note when node_type_* fields set** After upgrading to a version that supports data tiers (7.10.0 or above), the `node_type_*` has no effect even if specified. The provider automatically migrates the `node_type_*` fields to the appropriate `node_roles` as set by the deployment template. After having upgraded to `7.10.0` or above, the fields should be removed from the terraform configuration, if explicitly configured.
 
+##### Autoscaling
+
+The optional `elasticsearch.autoscaling` block supports the following arguments:
+
+* `min_size` - (Optional) Defines the minimum size the deployment will scale down to. When set, scale down will be enabled, please note that not all the tiers support this option.
+* `min_size_resource` - (Optional) Defines the resource type the scale down will use (Defaults to `"memory"`).
+* `max_size` - (Optional) Defines the maximum size the deployment will scale up to. When set, scaling up will be enabled. All tiers should support this option.
+* `max_size_resource` - (Optional) Defines the resource type the scale up will use (Defaults to `"memory"`).
+
+-> Note that none of these settings will take effect unless `elasticsearch.autoscale` is set to `"true"`.
+
+Please refer to the [Deployment Autoscaling](https://www.elastic.co/guide/en/cloud/current/ec-autoscaling.html) documentation for an updated list of the Elasticsearch tiers supporting scale up and scale down.
+
 ##### Config
 
 The optional `elasticsearch.config` block supports the following arguments:
@@ -391,6 +413,7 @@ In addition to all the arguments above, the following attributes are exported:
 * `elasticsearch.#.topology.#.node_type_ingest` - Node type (ingest) for the Elasticsearch topology element.
 * `elasticsearch.#.topology.#.node_type_ml` - Node type (machine learning) for the Elasticsearch topology element.
 * `elasticsearch.#.topology.#.node_roles` - List of roles for the topology element. They are inferred from the deployment template.
+* `elasticsearch.#.topology.#.autoscaling.#.policy_override_json` - Computed policy overrides set directly via the API or other clients.
 * `elasticsearch.#.snapshot_source.#.source_elasticsearch_cluster_id` - ID of the Elasticsearch cluster that will be used as the source of the snapshot.
 * `elasticsearch.#.snapshot_source.#.snapshot_name` - Name of the snapshot to restore.
 * `kibana.#.resource_id` - Kibana resource unique identifier.

ec/acc/deployment_autoscaling_test.go

@@ -0,0 +1,112 @@
+// Licensed to Elasticsearch B.V. under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Elasticsearch B.V. licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package acc
+
+import (
+	"testing"
+
+	"github.com/hashicorp/terraform-plugin-sdk/v2/helper/acctest"
+	"github.com/hashicorp/terraform-plugin-sdk/v2/helper/resource"
+)
+
+func TestAccDeployment_autoscaling(t *testing.T) {
+	resName := "ec_deployment.autoscaling"
+	randomName := prefix + acctest.RandStringFromCharSet(10, acctest.CharSetAlphaNum)
+	startCfg := "testdata/deployment_autoscaling_1.tf"
+	disableAutoscale := "testdata/deployment_autoscaling_2.tf"
+
+	cfgF := func(cfg string) string {
+		return fixtureAccDeploymentResourceBasic(
+			t, cfg, randomName, getRegion(), defaultTemplate,
+		)
+	}
+
+	resource.ParallelTest(t, resource.TestCase{
+		PreCheck:          func() { testAccPreCheck(t) },
+		ProviderFactories: testAccProviderFactory,
+		CheckDestroy:      testAccDeploymentDestroy,
+		Steps: []resource.TestStep{
+			{
+				Config: cfgF(startCfg),
+				Check: resource.ComposeAggregateTestCheckFunc(
+					resource.TestCheckResourceAttr(resName, "elasticsearch.#", "1"),
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.autoscale", "true"),
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.#", "3"),
+					resource.TestCheckResourceAttrSet(resName, "elasticsearch.0.topology.0.instance_configuration_id"),
+
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.id", "hot_content"),
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.size", "1g"),
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.size_resource", "memory"),
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.zone_count", "1"),
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.autoscaling.#", "1"),
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.autoscaling.0.max_size", "8g"),
+
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.1.id", "ml"),
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.1.size", "1g"),
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.1.size_resource", "memory"),
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.1.zone_count", "1"),
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.1.autoscaling.#", "1"),
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.1.autoscaling.0.max_size", "4g"),
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.1.autoscaling.0.min_size", "1g"),
+
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.2.id", "warm"),
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.2.size", "2g"),
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.2.size_resource", "memory"),
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.2.zone_count", "1"),
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.2.autoscaling.#", "1"),
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.2.autoscaling.0.max_size", "15g"),
+
+					resource.TestCheckResourceAttr(resName, "kibana.#", "0"),
+					resource.TestCheckResourceAttr(resName, "apm.#", "0"),
+					resource.TestCheckResourceAttr(resName, "enterprise_search.#", "0"),
+				),
+			},
+			// also disables ML
+			{
+				Config: cfgF(disableAutoscale),
+				// When disabling a tier the plan will be non empty on refresh
+				// since the topology block is present with size = "0g".
+				ExpectNonEmptyPlan: true,
+				Check: resource.ComposeAggregateTestCheckFunc(
+					resource.TestCheckResourceAttr(resName, "elasticsearch.#", "1"),
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.autoscale", "false"),
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.#", "2"),
+					resource.TestCheckResourceAttrSet(resName, "elasticsearch.0.topology.0.instance_configuration_id"),
+
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.id", "hot_content"),
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.size", "1g"),
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.size_resource", "memory"),
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.zone_count", "1"),
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.autoscaling.#", "1"),
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.autoscaling.0.max_size", "8g"),
+
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.1.id", "warm"),
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.1.size", "2g"),
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.1.size_resource", "memory"),
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.1.zone_count", "1"),
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.1.autoscaling.#", "1"),
+					resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.1.autoscaling.0.max_size", "15g"),
+
+					resource.TestCheckResourceAttr(resName, "kibana.#", "0"),
+					resource.TestCheckResourceAttr(resName, "apm.#", "0"),
+					resource.TestCheckResourceAttr(resName, "enterprise_search.#", "0"),
+				),
+			},
+		},
+	})
+}

ec/acc/deployment_pre_node_role_migration_test.go

@@ -18,7 +18,6 @@
 package acc
 
 import (
-	"regexp"
 	"testing"
 
 	"github.com/hashicorp/terraform-plugin-sdk/v2/helper/acctest"
@@ -30,8 +29,7 @@ func TestAccDeployment_pre_node_roles(t *testing.T) {
 	randomName := prefix + acctest.RandStringFromCharSet(10, acctest.CharSetAlphaNum)
 	startCfg := "testdata/deployment_pre_node_roles_migration_1.tf"
 	upgradeVersionCfg := "testdata/deployment_pre_node_roles_migration_2.tf"
-	invalidWarmTopologyCfg := "testdata/deployment_pre_node_roles_migration_3.tf"
-	addWarmTopologyCfg := "testdata/deployment_pre_node_roles_migration_4.tf"
+	addWarmTopologyCfg := "testdata/deployment_pre_node_roles_migration_3.tf"
 
 	cfgF := func(cfg string) string {
 		return fixtureAccDeploymentResourceBasic(
@@ -84,10 +82,6 @@ func TestAccDeployment_pre_node_roles(t *testing.T) {
 					resource.TestCheckResourceAttr(resName, "enterprise_search.#", "0"),
 				),
 			},
-			{
-				Config:      cfgF(invalidWarmTopologyCfg),
-				ExpectError: regexp.MustCompile(`(?m)invalid configuration: 1 error occurred:\n\t\* elasticsearch topology warm: size cannot be zero\n\n`),
-			},
 			{
 				Config: cfgF(addWarmTopologyCfg),
 				Check: resource.ComposeAggregateTestCheckFunc(

ec/acc/testdata/deployment_autoscaling_1.tf

@@ -0,0 +1,41 @@
+data "ec_stack" "autoscaling" {
+  version_regex = "latest"
+  region        = "%s"
+}
+
+resource "ec_deployment" "autoscaling" {
+  name                   = "%s"
+  region                 = "%s"
+  version                = data.ec_stack.autoscaling.version
+  deployment_template_id = "%s"
+
+  elasticsearch {
+    autoscale = "true"
+
+    topology {
+      id         = "hot_content"
+      size       = "1g"
+      zone_count = 1
+      autoscaling {
+        max_size = "8g"
+      }
+    }
+    topology {
+      id         = "ml"
+      size       = "1g"
+      zone_count = 1
+      autoscaling {
+        min_size = "1g"
+        max_size = "4g"
+      }
+    }
+    topology {
+      id         = "warm"
+      size       = "2g"
+      zone_count = 1
+      autoscaling {
+        max_size = "15g"
+      }
+    }
+  }
+}

ec/acc/testdata/deployment_autoscaling_2.tf

@@ -0,0 +1,41 @@
+data "ec_stack" "autoscaling" {
+  version_regex = "latest"
+  region        = "%s"
+}
+
+resource "ec_deployment" "autoscaling" {
+  name                   = "%s"
+  region                 = "%s"
+  version                = data.ec_stack.autoscaling.version
+  deployment_template_id = "%s"
+
+  elasticsearch {
+    autoscale = "false"
+
+    topology {
+      id         = "hot_content"
+      size       = "1g"
+      zone_count = 1
+      autoscaling {
+        max_size = "8g"
+      }
+    }
+    topology {
+      id         = "ml"
+      size       = "0g"
+      zone_count = 1
+      autoscaling {
+        min_size = "0g"
+        max_size = "4g"
+      }
+    }
+    topology {
+      id         = "warm"
+      size       = "2g"
+      zone_count = 1
+      autoscaling {
+        max_size = "15g"
+      }
+    }
+  }
+}

ec/acc/testdata/deployment_pre_node_roles_migration_3.tf

@@ -17,6 +17,7 @@ resource "ec_deployment" "pre_nr" {
     }
     topology {
       id         = "warm"
+      size       = "2g"
       zone_count = 1
     }
   }

ec/ecdatasource/deploymentdatasource/datasource.go

@@ -110,7 +110,10 @@ func modelToState(d *schema.ResourceData, res *models.DeploymentGetResponse) err
 		}
 	}
 
-	elasticsearchFlattened := flattenElasticsearchResources(res.Resources.Elasticsearch)
+	elasticsearchFlattened, err := flattenElasticsearchResources(res.Resources.Elasticsearch)
+	if err != nil {
+		return err
+	}
 	if err := d.Set("elasticsearch", elasticsearchFlattened); err != nil {
 		return err
 	}