Elasticsearch

What does this do?

Creates an Elastic Cloud Deployment. By default, it includes traffic filter rules to only allow traffic from allowed IPs and the VPC of the accompanying GCP project.

How to provision this module?

1. Add the module with every variable you want to set except for the private_service_connect_id variable and make/merge a PR with these settings.
   • If you are going to make multiple instances of this module in the same GCP project/region, be sure to set the variable project_name to differentiate resources in both GCP and Elastic Cloud.
   • If you want to make the deployment publicly accessible, set make_public to true.
   • The variable allowed_ips should be set to a map where the values are all IP addresses/CIDR ranges that should be granted access to the Deployment. This needs to be set if make_public is false (which is the default value).
   • If you do not need to use a Private Service Connect endpoint, you can set disable_psc to true. If you do this, you do not need to follow steps 2 and 3.
2. Follow the steps here to create a Private Service Connect endpoint in the proper GCP project.
   • For target please select Published Service from the radio buttons
   • For target service use the corresponding URI from this page depending on the region of the project.
   • For endpoint name please use the naming convention <GCP_PROJECT_NAME>-psc.
   • For subnetwork please select the subnet where the GKE cluster resides.
   • For IP address select <GCP_PROJECT_ID>-psc-address (or <PROJECT_NAME>-psc-address if you set the variable project_name) from the dropdown.
   • For service directory please do NOT choose a namespace as this will place the PSC in the default namespace (which is the desired bahavior)
3. Once the private service connect is created, grab the PSC Connection ID and update the private_service_connect_id variable in the module and raise a second PR with these updates.

Anonymous access

If the deployment is not publicly accessible and the variable enable_anonymous_access is set to true, then anonymous access (with limited permissions) is enabled and all users will have to do to access the elasticsearch cluster is to curl the endpoint, which is conveniently provided as an output variable. If they have need of greater permissions, or are trying to access a publicly available deployment, then they will need to use the username/password of the cluster to gain access. These are available as outputs as well, however, the password is marked as sensitive and end users will not be able to directly access it.

Usage Examples

Regular Use

module "elasticsearch" {
  source     = "[email protected]:dapperlabs-platform/terraform-elastic-cloud-gcp.git?ref=<VERSION>"
  project_id = module.project.project_id
  region     = var.default_region

  allowed_ips = <MAP_OF_IPS> # A map object where the values in the key:value pairs are IPs or CIDR ranges

  // private_service_connect_id = "<PRIVATE_SERVICE_CONNECT_ID>" # Uncomment this after manually creating Private Service Connect
}

Deploying multiple times in single environment example

module "elasticsearch" {
  source       = "[email protected]:dapperlabs-platform/terraform-elastic-cloud-gcp.git?ref=<VERSION>"
  project_id   = module.project.project_id
  project_name = "<PROJECT_NAME>"
  region       = var.default_region

  allowed_ips = <MAP_OF_IPS> # A map object where the values in the key:value pairs are IPs or CIDR ranges

  // private_service_connect_id = "<PRIVATE_SERVICE_CONNECT_ID>" # Uncomment this after manually creating Private Service Connect
}

Custom Elastic Deployment Needs

module "elasticsearch" {
  source     = "[email protected]:dapperlabs-platform/terraform-elastic-cloud-gcp.git?ref=<VERSION>"
  project_id = module.project.project_id
  region     = var.default_region

  allowed_ips = <MAP_OF_IPS> # A map object where the values in the key:value pairs are IPs or CIDR ranges

  elastic_topology = [
    {
      id         = "hot_content"
      size       = "16g"
      zone_count = 6
      autoscaling = {
        max_size = "128g"
      }
    },
    {
      id         = "cold"
      size       = "1g"
      zone_count = 2
      autoscaling = {
        max_size = "16g"
      }
    }
  ]

  // private_service_connect_id = "<PRIVATE_SERVICE_CONNECT_ID>" # Uncomment this after manually creating Private Service Connect
}

Updating this module

make updates the README.md file based on Terraform changes.

Requires

1. terraform Download Brew
2. terraform-docs to update the README. Download Brew
3. make to update the README. Download

Requirements

Name	Version
ec	~> 0.4.0
elasticstack	~> 0.3.3
google	~> 4.0
google-beta	~> 4.0

Providers

Name	Version
ec	~> 0.4.0
elasticstack	~> 0.3.3
google	~> 4.0

Modules

No modules.

Resources

Name	Type
ec_deployment.elastic_cloud_deployment	resource
ec_deployment_traffic_filter.filter_allowed_ips	resource
ec_deployment_traffic_filter.filter_gcp_psc	resource
elasticstack_elasticsearch_security_role.anonymous_role	resource
google_compute_address.psc_address	resource
google_dns_managed_zone.psc_managed_zone	resource
google_dns_record_set.psc_managed_zone_record	resource
ec_stack.version	data source
google_compute_address.psc_address	data source
google_compute_network.network	data source

Inputs

Name	Description	Type	Default	Required
allowed_ips	Map of IPs to allowlist for access to Elastic Cloud deployment. If make_public is false, this variable must be set or you will not be able to access the deployment.	map(string)	null	no
disable_psc	This will disable the creation of the networking resources required to provide authentication to Elastic Cloud via GCP Private Service connect	bool	false	no
elastic_deployment_template_name	The instance type to use in the deployment. Go to https://www.elastic.co/guide/en/cloud/current/ec-regions-templates-instances.html for a list of options.	string	"gcp-general-purpose"	no
elastic_topology	Configuration settings list for desired Elasticsearch topologies. See https://registry.terraform.io/providers/elastic/ec/latest/docs/resources/ec_deployment#topology for definitions of topology settings.	list(object({ id = string size = string zone_count = string autoscaling = object({ max_size = string }) }))	[ { "autoscaling": { "max_size": "64g" }, "id": "hot_content", "size": "4g", "zone_count": 3 } ]	no
elastic_version_regex	The regex of the version of Elasticsearch to use. This will be used in a data object that queries available versions on Elastic Cloud and will return the most recent version that matches the constraints. This will only be used on the first apply and then the version will be locked in place so as to not cause unexpected cluster upgrades. CAUTION: Elastic Cloud only supports 3 versions at any one time: the version, the latest version of the previous minor version, and the latest version of the previous major version. The default is latest as this is the safeest way to ensure you will always deploy with a supported version of Elasticsearch. But you can enter in a regex if you have a specific version you need, just keep in mind that very few versions are supported.	string	"latest"	no
enable_anonymous_access	This will enable users to access the cluster anonymously (i.e. without username/password). Will only be enabled for private deployments.	bool	false	no
kibana_topology	Configuration settings list for desired Kibana topologies. See https://registry.terraform.io/providers/elastic/ec/latest/docs/resources/ec_deployment#topology for definitions of topology settings.	list(object({ size = string zone_count = string }))	[]	no
make_public	Boolean that will expose the Elastic Cloud deployments to the public internet. You will still need a username/password to connect	bool	false	no
private_service_connect_id	The private service connect id, add this after manually creating the private service connect in the GCP console	string	"0"	no
project_id	The GCP project id.	string	n/a	yes
project_name	A name variable used to name the resources. Should only be set if deploying to a Gen 2 project	string	null	no
region	Region for the deployment	string	"us-west1"	no
request_id	This variable is sometimes needed when the Elastic API encounters an error. Only set this if told to by the output of a Terraform apply	string	null	no
vpc_name	The name of the VPC network of the GKE cluster we want to allow communication from	string	"gke-application-cluster-vpc"	no

Outputs

Name	Description
elasticsearch_https_endpoint	n/a
elasticsearch_password	n/a
elasticsearch_user	n/a
kibana_endpoint	n/a