Add SSL/TLS support

.kitchen.yml

@@ -132,3 +132,7 @@ suites:
     provisioner:
       playbook: test/integration/issue-test.yml
       idempotency_test: false
+  - name: xpack-upgrade-trial
+    provisioner:
+      playbook: test/integration/xpack-upgrade-trial.yml
+      idempotency_test: false

README.md

@@ -24,6 +24,7 @@ The latest Elasticsearch versions of 7.x & 6.x are actively tested.
 * For multi-instances use cases, we are now recommending Docker containers using our official images (https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html).
 
 ## Dependency
+
 This role uses the json_query filter which [requires jmespath](https://github.com/ansible/ansible/issues/24319) on the local machine.
 
 ## Usage
@@ -69,6 +70,10 @@ This playbook uses [Kitchen](https://kitchen.ci/) for CI and local testing.
 
 ### Running the tests
 
+* Ensure you have checked out this repository to `elaticsearch`, not `ansible-elasticsearch`.
+* If you don't have a Gold or Platinum license to test with you can run the trial versions of the `xpack-upgrade` and `issue-test` suites by appending `-trial` to the `PATTERN` variable.
+* You may need to explicity specify `VERSION=7.x` if some suites are failing.
+
 Install the ruby dependencies with bundler
 
 ```sh
@@ -151,12 +156,13 @@ Whilst the role installs Elasticsearch with the default configuration parameters
 * ```es_config['http.port']``` - the http port for the node
 * ```es_config['transport.port']``` - the transport port for the node
 * ```es_config['discovery.seed_hosts']``` - the unicast discovery list, in the comma separated format ```"<host>:<port>,<host>:<port>"``` (typically the clusters dedicated masters)
+* ```es_config['cluster.initial_master_nodes']``` - for 7.x and above the list of master-eligible nodes to boostrap the cluster, in the comma separated format ```"<node.name>:<port>,<node.name>:<port>"``` (typically the node names of the clusters dedicated masters)
 * ```es_config['network.host']``` - sets both network.bind_host and network.publish_host to the same host value. The network.bind_host setting allows to control the host different network components will bind on.
 
-The network.publish_host setting allows to control the host the node will publish itself within the cluster so other nodes will be able to connect to it.
+The `network.publish_host` setting allows to control the host the node will publish itself within the cluster so other nodes will be able to connect to it.
 
 See https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html for further details on default binding behaviour and available options.
-The role makes no attempt to enforce the setting of these are requires users to specify them appropriately.  IT is recommended master nodes are listed and thus deployed first where possible.
+The role makes no attempt to enforce the setting of these are requires users to specify them appropriately.  It is recommended master nodes are listed and thus deployed first where possible.
 
 A more complex example:
 
@@ -262,7 +268,7 @@ ansible-playbook -i hosts ./your-playbook.yml
 X-Pack features, such as Security, are supported.
 
 The parameter `es_xpack_features` allows to list xpack features to install (example: `["alerting","monitoring","graph","security","ml"]`).
-When the list is empty, it install all features available with the current licence.
+When the list is empty, it installs all features available with the current licence.
 
 * ```es_role_mapping``` Role mappings file declared as yml as described [here](https://www.elastic.co/guide/en/x-pack/current/mapping-roles.html)
 
@@ -347,6 +353,8 @@ es_roles:
 es_xpack_license: "{{ lookup('file', playbook_dir + '/files/' + es_cluster_name + '/license.json') }}"
 ```
 
+If you don't have a license you can enable the 30-day trial by setting `es_xpack_trial` to `true`.
+
 X-Pack configuration parameters can be added to the elasticsearch.yml file using the normal `es_config` parameter.
 
 For a full example see [here](https://github.com/elastic/ansible-elasticsearch/blob/master/test/integration/xpack-upgrade.yml)
@@ -360,26 +368,35 @@ In order for native users and roles to be configured, the role calls the Elastic
 
 These can either be set to a user declared in the file based realm, with admin permissions, or the default "elastic" superuser (default password is changeme).
 
+#### X-Pack Security SSL/TLS
+
+* To configure your cluster with SSL/TLS for HTTP and/or transport communications follow the [SSL/TLS setup procedure](./docs/ssl-tls-setup.md)
+
 
 ### Additional Configuration
 
 In addition to es_config, the following parameters allow the customization of the Java and Elasticsearch versions as well as the role behaviour. Options include:
 
 * ```es_enable_xpack```  Default `true`. Setting this to `false` will install the oss release of elasticsearch
+* `es_xpack_trial` Default `false`. Setting this to `true` will start the 30-day trail once the cluster starts.
 * ```es_version``` (e.g. "7.4.1").
 * ```es_api_host``` The host name used for actions requiring HTTP e.g. installing templates. Defaults to "localhost".
 * ```es_api_port``` The port used for actions requiring HTTP e.g. installing templates. Defaults to 9200. **CHANGE IF THE HTTP PORT IS NOT 9200**
 * ```es_api_basic_auth_username``` The Elasticsearch username for making admin changing actions. Used if Security is enabled. Ensure this user is admin.
 * ```es_api_basic_auth_password``` The password associated with the user declared in `es_api_basic_auth_username`
+* `es_delete_unmanaged_file` Default `true`. Set to false to keep file realm users that have been added outside of ansible.
+* `es_delete_unmanaged_native` Default `true`. Set to false to keep native realm users that have been added outside of ansible.
 * ```es_start_service``` (true (default) or false)
 * ```es_plugins_reinstall``` (true or false (default) )
 * ```es_plugins``` an array of plugin definitions e.g.:
+
   ```yaml
     es_plugins:
       - plugin: ingest-attachment
   ```
+
 * ```es_path_repo``` Sets the whitelist for allowing local back-up repositories
-* ```es_action_auto_create_index ``` Sets the value for auto index creation, use the syntax below for specifying indexes (else true/false):
+* ```es_action_auto_create_index``` Sets the value for auto index creation, use the syntax below for specifying indexes (else true/false):
      es_action_auto_create_index: '[".watches", ".triggered_watches", ".watcher-history-*"]'
 * ```es_allow_downgrades``` For development purposes only. (true or false (default) )
 * ```es_java_install``` If set to true, Java will be installed. (false (default for 7.x) or true (default for 6.x))
@@ -397,6 +414,7 @@ Earlier examples illustrate the installation of plugins using `es_plugins`.  For
 If installing Monitoring or Alerting, ensure the license plugin is also specified.  Security configuration currently has limited support, but more support is planned for later versions.
 
 To configure X-pack to send mail, the following configuration can be added to the role. When require_auth is true, you will also need to provide the user and password. If not these can be removed:
+
 ```yaml
     es_mail_config:
         account: <functional name>
@@ -445,7 +463,7 @@ To define proxy only for a particular plugin during its installation:
 * The playbook relies on the inventory_name of each host to ensure its directories are unique
 * KitchenCI has been used for testing.  This is used to confirm images reach the correct state after a play is first applied.  We currently test the latest version of 7.x and 6.x on all supported platforms.
 * The role aims to be idempotent.  Running the role multiple times, with no changes, should result in no state change on the server.  If the configuration is changed, these will be applied and Elasticsearch restarted where required.
-* In order to run x-pack tests a license file with security enabled is required. A trial license is appropriate. Set the environment variable `ES_XPACK_LICENSE_FILE` to the full path of the license file prior to running tests.
+* In order to run x-pack tests a license file with security enabled is required. Set the environment variable `ES_XPACK_LICENSE_FILE` to the full path of the license file prior to running tests. A trial license is appropriate and can be used by setting `es_xpack_trial` to `true`
 
 ## IMPORTANT NOTES RE PLUGIN MANAGEMENT
 

ansible.cfg

@@ -1 +1 @@
-[defaults]
+[defaults]

defaults/main.yml

@@ -33,11 +33,34 @@ es_max_threads: 8192
 es_max_map_count: 262144
 es_allow_downgrades: false
 es_xpack_features: []
+es_xpack_trial: false
 #These are used for internal operations performed by ansible.
 #They do not affect the current configuration
+es_api_scheme: "http"
 es_api_host: "localhost"
 es_api_port: 9200
+es_api_uri: "{{ es_api_scheme }}://{{ es_api_host }}:{{ es_api_port }}"
+es_security_api: "{{ '_security' if es_version is version_compare('7.0.0', '>=') else '_xpack/security' }}"
+es_license_api: "{{ '_license' if es_version is version_compare('7.0.0', '>=') else '_xpack/license' }}"
+es_api_sleep: 15
 es_debian_startup_timeout: 10
 
 # JVM custom parameters
 es_jvm_custom_parameters: ''
+
+# SSL/TLS parameters
+es_enable_http_ssl: false
+es_enable_transport_ssl: false
+es_ssl_keystore: ""
+es_ssl_keystore_password: ""
+es_ssl_truststore: ""
+es_ssl_truststore_password: ""
+es_ssl_key: ""
+es_ssl_key_password: ""
+es_ssl_certificate: ""
+es_ssl_certificate_authority: ""
+es_ssl_certificate_path: "{{ es_conf_dir }}/certs"
+es_ssl_verification_mode: "certificate"
+es_validate_certs: "yes"
+es_delete_unmanaged_file: true
+es_delete_unmanaged_native: true

docs/ssl-tls-setup.md

@@ -0,0 +1,75 @@
+# X-Pack Security SSL/TLS
+
+The role allows configuring HTTP and transport layer SSL/TLS for the cluster. You will need to generate and provide your own PKCS12 or PEM encoded certificates as described in [Encrypting communications in Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/7.4/configuring-tls.html#configuring-tls).
+
+The following should be configured to ensure a security-enabled cluster successfully forms:
+
+* `es_enable_http_ssl`  Default `false`. Setting this to `true` will enable HTTP client SSL/TLS
+* `es_enable_transport_ssl` - Default `false`. Setting this to `true` will enable transport layer SSL/TLS
+
+When using a [PKCS12](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-settings.html#security-http-pkcs12-files) keystore and truststore:
+
+* `es_ssl_keystore`  path to your PKCS12 keystore (can be the same as `es_ssl_truststore`)
+* `es_ssl_keystore_password`  set this if your keystore is protected with a password
+* `es_ssl_truststore`  path to your PKCS12 keystore (can be the same as `es_ssl_keystore`)
+* `es_ssl_truststore_password`  set this if your truststore is protected with a password
+
+When using [PEM encoded](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-settings.html#_pem_encoded_files_3) certificates:
+
+* `es_ssl_key`  path to your SSL key
+* `es_ssl_key_password`  set this if your SSL key is protected with a password
+* `es_ssl_certificate`  the path to your SSL certificate
+
+## Generating an SSL keystore
+
+With a password:
+
+```shell
+$ bin/elasticsearch-certutil ca --out ./my-ca.p12 --pass "ca_password"
+$ bin/elasticsearch-certutil cert --ca ./my-ca.p12 --ca-pass "ca_password" --out ./my-keystore.p12 --pass "keystore_password"
+```
+
+Without a password:
+
+```shell
+$ bin/elasticsearch-certutil ca --out ./my-ca.p12 --pass ""
+$ bin/elasticsearch-certutil cert --ca ./my-ca.p12 --out ./my-keystore.p12 --pass ""
+```
+
+## Additional optional SSL/TLS configuration
+
+* `es_ssl_certificate_path`  Default `{{ es_conf_dir }}/certs`. The location where certificates should be stored on the ES node.
+* `es_ssl_verification_mode` Default `certificate`. See [SSL verification_mode](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-settings.html#ssl-tls-settings) for options.
+* `es_ssl_certificate_authority`  PEM encoded certificate file that should be trusted.
+* `es_validate_certs`  Default `yes`. Determines if ansible should validate SSL certificates when performing actions over HTTPS. e.g. installing templates and managing native users.
+
+## Example SSL/TLS configuration
+
+```yaml
+- name: Elasticsearch with SSL/TLS enabled
+  hosts: localhost
+  roles:
+    - role: elastic.elasticsearch
+  vars:
+    es_config:
+      node.name: "node1"
+      cluster.name: "custom-cluster"
+      discovery.seed_hosts: "localhost:9301"
+      http.port: 9201
+      transport.port: 9301
+      node.data: false
+      node.master: true
+      bootstrap.memory_lock: true
+      xpack.security.authc.realms.file.file1.order: 0
+      xpack.security.authc.realms.native.native1.order: 1
+    es_heap_size: 1g
+    es_api_basic_auth_username: elastic
+    es_api_basic_auth_password: changeme
+    es_enable_http_ssl: true
+    es_enable_transport_ssl: true
+    es_ssl_keystore: "files/certs/my-keystore.p12"
+    es_ssl_truststore: "files/certs/my-truststore.p12"
+    es_ssl_keystore_password: "keystore_password"
+    es_ssl_truststore_password: "truststore_password"
+    es_validate_certs: no
+```

tasks/compatibility-variables.yml

@@ -23,3 +23,8 @@
     es_other_package_name: "elasticsearch"
   when:
     - not es_enable_xpack
+
+- name: Set the URL scheme to https if SSL/TLS is enabled
+  set_fact:
+    es_api_scheme: "https"
+  when: es_enable_http_ssl | bool

tasks/elasticsearch-parameters.yml

@@ -17,6 +17,13 @@
     - es_api_basic_auth_username is not defined
     - es_api_basic_auth_password is not defined
 
+- name: fail when ssl enabled without defining a key and certificate
+  fail: msg="Enabling SSL/TLS (es_enable_http_ssl or es_enable_transport_ssl) requires es_ssl_keystore and es_ssl_truststore or es_ssl_key and es_ssl_certificate to be provided"
+  when:
+    - es_enable_http_ssl or es_enable_transport_ssl
+    - (es_ssl_key == "" or es_ssl_certificate == "")
+    - (es_ssl_keystore == "" or es_ssl_truststore == "")
+
 - name: set fact file_reserved_users
   set_fact: file_reserved_users={{ es_users.file.keys() | list | intersect (reserved_xpack_users) }}
   when: es_users is defined and es_users.file is defined and (es_users.file.keys() | list | length > 0) and (es_users.file.keys() | list | intersect (reserved_xpack_users) | length > 0)

tasks/elasticsearch-ssl.yml

@@ -0,0 +1,112 @@
+---
+
+- name: set fact es_same_keystore
+  set_fact: es_same_keystore=false
+
+- name: set fact es_same_keystore if stores match
+  set_fact: es_same_keystore=true
+  when: es_ssl_keystore == es_ssl_truststore
+
+- name: ensure certificate directory exists
+  file:
+    dest: "{{ es_ssl_certificate_path }}"
+    state: directory
+    owner: root
+    group: "{{ es_group }}"
+    mode: 0750
+
+- name: Upload SSL/TLS keystore
+  copy:
+    src: "{{ es_ssl_keystore }}"
+    dest: "{{ es_ssl_certificate_path }}/{{ es_ssl_keystore | basename }}"
+    owner: "{{ es_user }}"
+    group: "{{ es_group }}"
+    mode: 0640
+  when: es_ssl_keystore and es_ssl_truststore
+  notify: restart elasticsearch
+  register: copy_keystore
+
+- name: Upload SSL/TLS truststore
+  copy:
+    src: "{{ es_ssl_truststore }}"
+    dest: "{{ es_ssl_certificate_path }}/{{ es_ssl_truststore | basename }}"
+    owner: "{{ es_user }}"
+    group: "{{ es_group }}"
+    mode: 0640
+  when: es_ssl_keystore and es_ssl_truststore
+  notify: restart elasticsearch
+  register: copy_truststore
+
+- name: Upload SSL/TLS key and certificate
+  copy:
+    src: "{{ item }}"
+    dest: "{{ es_ssl_certificate_path }}/{{ item | basename }}"
+    owner: "{{ es_user }}"
+    group: "{{ es_group }}"
+    mode: 0640
+  with_items:
+    - "{{ es_ssl_key }}"
+    - "{{ es_ssl_certificate }}"
+  when: es_ssl_key and es_ssl_certificate
+  #Restart if these change
+  notify: restart elasticsearch
+  register: copy_certificates
+
+- name: Upload SSL Certificate Authority
+  copy:
+    src: "{{ es_ssl_certificate_authority }}"
+    dest: "{{ es_ssl_certificate_path }}/{{ es_ssl_certificate_authority | basename }}"
+    owner: "{{ es_user }}"
+    group: "{{ es_group }}"
+    mode: 0640
+  #Restart if this changes
+  notify: restart elasticsearch
+  when: es_ssl_certificate_authority | bool
+
+- name: Set keystore password
+  shell: echo "{{ es_ssl_keystore_password }}" | {{ es_home }}/bin/elasticsearch-keystore add -x -f 'xpack.security.{{ item }}.ssl.keystore.secure_password'
+  no_log: True
+  when: es_ssl_keystore_password and (copy_keystore.changed or (es_same_keystore and copy_truststore.changed))
+  with_items:
+    - http
+    - transport
+
+- name: Set truststore password
+  shell: echo "{{ es_ssl_truststore_password }}" | {{ es_home }}/bin/elasticsearch-keystore add -x -f 'xpack.security.{{ item }}.ssl.truststore.secure_password'
+  no_log: True
+  when: es_ssl_truststore_password and (copy_truststore.changed or (es_same_keystore and copy_keystore.changed))
+  with_items:
+    - http
+    - transport
+
+- name: Remove keystore password
+  shell: "{{ es_home }}/bin/elasticsearch-keystore remove 'xpack.security.{{ item }}.ssl.keystore.secure_password'"
+  when: es_ssl_keystore_password == "" and (copy_keystore.changed or (es_same_keystore and copy_truststore.changed))
+  ignore_errors: yes
+  with_items:
+    - http
+    - transport
+
+- name: Remove truststore password
+  shell: "{{ es_home }}/bin/elasticsearch-keystore remove 'xpack.security.{{ item }}.ssl.truststore.secure_password'"
+  when: es_ssl_truststore_password == "" and (copy_truststore.changed or (es_same_keystore and copy_keystore.changed))
+  ignore_errors: yes
+  with_items:
+    - http
+    - transport
+
+- name: Set key password
+  shell: echo "{{ es_ssl_key_password }}" | {{ es_home }}/bin/elasticsearch-keystore add -x -f 'xpack.security.{{ item }}.ssl.secure_key_passphrase'
+  no_log: True
+  when: es_ssl_key_password and copy_certificates.changed
+  with_items:
+    - http
+    - transport
+
+- name: Remove key password
+  shell: "{{ es_home }}/bin/elasticsearch-keystore remove 'xpack.security.{{ item }}.ssl.secure_key_passphrase'"
+  when: es_ssl_key_password == "" and copy_certificates.changed
+  ignore_errors: yes
+  with_items:
+    - http
+    - transport

tasks/elasticsearch-template.yml

@@ -16,14 +16,15 @@
 
 - name: Install templates
   uri:
-    url: "http://{{es_api_host}}:{{es_api_port}}/_template/{{item | filename}}"
+    url: "{{ es_api_uri }}/_template/{{item | filename}}"
     method: PUT
     status_code: 200
     user: "{{es_api_basic_auth_username | default(omit)}}"
     password: "{{es_api_basic_auth_password | default(omit)}}"
     force_basic_auth: yes
     body_format: json
     body: "{{ lookup('file', item) }}"
+    validate_certs: "{{ es_validate_certs }}"
   when: load_templates.changed and es_start_service
   with_fileglob:
     - "{{ es_templates_fileglob | default('') }}"