From abc006cd426aee1ed42b4ecc27d25e87e79965fe Mon Sep 17 00:00:00 2001 From: Michael Schurter Date: Wed, 26 Jul 2017 17:25:02 -0700 Subject: [PATCH 01/13] WIP tls guide --- demo/vagrant/Vagrantfile | 3 +- website/source/guides/tls.html.md | 328 ++++++++++++++++++++++++++++++ 2 files changed, 330 insertions(+), 1 deletion(-) create mode 100644 website/source/guides/tls.html.md diff --git a/demo/vagrant/Vagrantfile b/demo/vagrant/Vagrantfile index 54b5109f9f8..1742b1ee14e 100644 --- a/demo/vagrant/Vagrantfile +++ b/demo/vagrant/Vagrantfile @@ -7,7 +7,8 @@ sudo apt-get update sudo DEBIAN_FRONTEND=noninteractive apt-get install -y unzip curl vim \ apt-transport-https \ ca-certificates \ - software-properties-common + software-properties-common \ + openssl # Download Nomad NOMAD_VERSION=0.6.0 diff --git a/website/source/guides/tls.html.md b/website/source/guides/tls.html.md new file mode 100644 index 00000000000..7135f94be63 --- /dev/null +++ b/website/source/guides/tls.html.md @@ -0,0 +1,328 @@ +--- +layout: "guides" +page_title: "Securing Nomad with TLS" +sidebar_current: "guides-tls" +description: |- + Securing Nomad's cluster communication with TLS is XXX TODO XXX +--- + +# Securing Nomad with TLS + +Securing Nomad's cluster communication is not only important for security but +can even ease operations by preventing mistakes and misconfigurations. Nomad +optionally uses mutual TLS (mTLS) for all HTTP and RPC communication. Nomad's +use of mTLS provides the following properties: + +* Prevent unauthorized Nomad access +* Prevent observing or tampering with Nomad communication +* Prevent client/server role or region misconfigurations + +The 3rd property is fairly unique to Nomad's use of TLS. While most uses of TLS +verify the identity of the server you're connecting to based on a domain name +such as `nomadproject.io`, Nomad verifies the node you're connecting to is in +the expected region and configured for the expected role (e.g. +`client.us-west.nomad`). + +Configuring TLS can be unfortunately complex process, but if you used the +[Getting Started guide's Vagrantfile][Vagrantfile] or have [OpenSSL][] and Nomad +installed this guide will provide you with a production ready TLS +configuration. + +~> Note that while Nomad's TLS configuration will be production ready, key + management and rotation is a complex subject not covered by this guide. + [Vault][] is the suggested solution for key generation and management. + +XXX TODO XXX - serf encryption key + +## Creating Certificates + +The first step to configuring TLS for Nomad is generating certificates. In +order to prevent unauthorized cluster access, Nomad requires all certificates +are signed by the sign Certificate Authority (CA). This should be a *private* +CA and not a public like [Let's Encrypt][letsencrypt] as any certificate signed +by this CA will be allowed to communicate with the cluster. + +### Certificate Authority + +You can generate a private CA certificate and key with OpenSSL: + +```shell +# Generate the CA's private key +# This file (nomad-ca.key) must be kept *secret* +openssl genrsa -out nomad-ca.key 4096 + +# Generate the CA's self-signed certicate +# This file (nomad-ca.crt) will be distributed to all nodes +openssl req -new -x509 -key nomad-ca.key -out nomad-ca.crt +You are about to be asked to enter information that will be incorporated +into your certificate request. +What you are about to enter is what is called a Distinguished Name or a DN. +There are quite a few fields but you can leave some blank +For some fields there will be a default value, +If you enter '.', the field will be left blank. +----- +Country Name (2 letter code) [AU]:. +State or Province Name (full name) [Some-State]:. +Locality Name (eg, city) []:. +Organization Name (eg, company) [Internet Widgits Pty Ltd]:. +Organizational Unit Name (eg, section) []:. +Common Name (e.g. server FQDN or YOUR name) []:Nomad CA +Email Address []:. +``` + +Your answers to OpenSSL's prompts are purely informational and not used by +Nomad. + +The CA key (`nomad-ca.key`) will be used to sign certificates for Nomad nodes +and must be kept private. The CA certificate (`nomad-ca.crt`) contains the +public key necessary to validate Nomad certificates and therefore must be +distributed to every node that requires access. + +### Node Certificates + +Once you have a CA certifacte and key you can generate and sign the +certificates Nomad will use directly. Traditionally TLS certificates use the +fully-qualified domain name of the system being identified as the certificate's +Common Name (CN). However, hosts (and therefore hostnames and IPs) are often +ephemeral in Nomad clusters. They come and go as clusters are scaled up and +down or outages occur. Not only would signing a new certificate per Nomad node +be difficult, but using a hostname provides no security or functional benefits +to Nomad. To fulfill the desired security properties (see above) Nomad +certificates are signed with their region and role such as: + +* `client.global.nomad` for a client node in the `global` region +* `server.us-west.nomad` for a server node in the `us-west` region + +To create certificates for the client and server in the cluster from the +[Getting Started guide][guide-cluster] with OpenSSL create the following +configuration file `nomad.conf`: + +```ini +basicConstraints = CA:FALSE +subjectAltName = @alt_names + +[alt_names] +DNS.1 = ${commonName} +DNS.2 = localhost +``` + +Using `localhost` as a subject alternate name (SAN) allows tools like `curl` to +be able to communicate with Nomad's HTTP API when run on the same host. Other +SANs may be added including a DNS resolvable hostname to allow remote HTTP +requests from third party tools. + +Then create client and server certificate and key pairs: + +```shell +# Client key and certificate +openssl genrsa -out client.global.nomad.key 4096 +openssl req -new -sha256 \ + -out client.global.nomad.csr \ + -key client.global.nomad.key \ + -subj /CN=client.global.nomad/ +openssl x509 -req \ + -in client.global.nomad.csr \ + -CA nomad-ca.crt \ + -CAkey nomad-ca.key \ + -days 3650 \ + -set_serial $(hexdump -e '"0x%x%x%x%x"' -n 16 /dev/urandom) \ + -extfile nomad.conf \ + -out client.global.nomad.crt + +# Server key and certificate +openssl genrsa -out server.global.nomad.key 4096 +openssl req -new -sha256 \ + -out server.global.nomad.csr \ + -key server.global.nomad.key \ + -subj /CN=server.global.nomad/ +openssl x509 -req \ + -in server.global.nomad.csr \ + -CA nomad-ca.crt \ + -CAkey nomad-ca.key \ + -days 3650 \ + -set_serial $(hexdump -e '"0x%x%x%x%x"' -n 16 /dev/urandom) \ + -extfile nomad.conf \ + -out server.global.nomad.crt +``` + +You should now have the following files: + +* `nomad-ca.key` - CA private key. Keep safe! +* `nomad-ca.crt` - CA public certificate. +* `client.global.nomad.key` - Nomad client node private key for the `global` region. +* `client.global.nomad.csr` - Nomad client node certificate signing request for the `global` region. +* `client.global.nomad.crt` - Nomad client node public certificate for the `global` region. +* `server.global.nomad.key` - Nomad server node private key for the `global` region. +* `server.global.nomad.csr` - Nomad server node certificate signing request for the `global` region. +* `server.global.nomad.crt` - Nomad server node public certificate for the `global` region. + +Each Nomad node should have the appropriate key (`.key`) and certificate +(`.crt`) file for its region and role. In addition each node needs the CA's +public certificate (`nomad-ca.crt`). + +## Configuring Nomad + +Once you have the appropriate key and certificates installed you're ready to +configure Nomad to use them for mTLS. Starting with the [server configuration +from the Getting Started guide][guide-server] add the following TLS specific +configuration options: + +```hcl +# Increase log verbosity +log_level = "DEBUG" + +# Setup data dir +data_dir = "/tmp/server1" + +# Enable the server +server { + enabled = true + + # Self-elect, should be 3 or 5 for production + bootstrap_expect = 1 +} + +# Require TLS +tls { + http = true + rpc = true + + ca_file = "nomad-ca.crt" + cert_file = "server.global.nomad.crt" + key_file = "server.global.nomad.key" + + verify_server_hostname = true + verify_https_client = true +} +``` + +The new `tls` section is worth breaking down in more detail: + +```hcl + http = true + rpc = true +``` + +This enables TLS for the HTTP and RPC protocols. Unlike web servers, Nomad +doesn't use separate ports for TLS and non-TLS traffic: your cluster should +either use TLS or not. + +```hcl + ca_file = "nomad-ca.crt" + cert_file = "server.global.nomad.crt" + key_file = "server.global.nomad.key" +``` + +The file lines should point to whereever you placed the certificate files on +the node. This guide assumes they're in Nomad's current directory. + +```hcl + verify_server_hostname = true + verify_https_client = true +``` + +These two settings are important for ensuring all of Nomad's mTLS security +properties are met. `verify_server_hostname` may be set to `false` to only +ensure that a node's certificate is signed by the same CA. This means any +service with a certificate from the same CA as Nomad can act as a client or +server of any region. + +`verify_https_client` may be disabled to allow non-Nomad clients (eg Consul or +curl) to communicate with the HTTPS API. + +~> Enabling `verify_https_client` feature effectively protects Nomad from + unauthorized network access at the cost of breaking compatibility with Consul + HTTPS health checks and third party tools like curl. + +### Client configuration + +The Nomad client configuration is similar with the only difference being the +certificate and key used: + +```hcl +# Increase log verbosity +log_level = "DEBUG" + +# Setup data dir +data_dir = "/tmp/client1" + +# Enable the client +client { + enabled = true + + # For demo assume we are talking to server1. For production, + # this should be like "nomad.service.consul:4647" and a system + # like Consul used for service discovery. + servers = ["127.0.0.1:4647"] +} + +# Modify our port to avoid a collision with server1 +ports { + http = 5656 +} + +# Require TLS +tls { + http = true + rpc = true + + ca_file = "nomad-ca.crt" + cert_file = "client.global.nomad.crt" + key_file = "client.global.nomad.key" + + verify_server_hostname = true + verify_https_client = true +} +``` + +### Running with TLS + +Now that we have certificates generated and configuration for a client and +server we can test our TLS-enabled cluster! + +In separate terminals start a server and client agent: + +```shell +# In one terminal... +nomad agent -config server1.hcl + +# ...and in another +nomad agent -config client1.hcl +``` + +Finally in a third terminal test out `nomad node-status`: + +```text +vagrant@nomad:~$ nomad node-status +Error querying node status: Get http://127.0.0.1:4646/v1/nodes: malformed HTTP response "\x15\x03\x01\x00\x02\x02" +``` + +Oh no! That didn't work! + +Don't worry, the Nomad CLI just defaults to `http://...` instead of +`https://...`. We can override this with an environment variable: + +```shell +export NOMAD_ADDR=https://localhost:4646 +export NOMAD_CACERT=nomad-ca.crt +export NOMAD_CLIENT_CERT=client.global.nomad.crt +``` + +The `NOMAD_CACERT` also needs to be set so the CLI can verify it's talking to +an actual Nomad node. Finally, the `NOMAD_CLIENT_CERT` needs to be set since we +enabled `verify_https_client` above which prevents any access lacking a client +certificate. Operators may wish to generate a certificate specifically for the +CLI as any certificate signed by Nomad's CA will work. + +XXX TODO XXX - an example of everything working + +## Switching an existing cluster to TLS + +XXX TODO XXX + +[guide-server]: https://raw.githubusercontent.com/hashicorp/nomad/master/demo/vagrant/server.hcl +[guide-cluster]: https://www.nomadproject.io/intro/getting-started/cluster.html +[letsencrypt]: https://letsencrypt.org/ +[OpenSSL]: https://www.openssl.org/ +[Vagrantfile]: https://raw.githubusercontent.com/hashicorp/nomad/master/demo/vagrant/Vagrantfile +[Vault]: https://www.vaultproject.io/docs/secrets/pki/index.html From a72e59a079696afaefedab4a42075e8c48fd63d8 Mon Sep 17 00:00:00 2001 From: Michael Schurter Date: Thu, 27 Jul 2017 10:38:44 -0700 Subject: [PATCH 02/13] Improve verify_https_clients wording --- website/source/guides/tls.html.md | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/website/source/guides/tls.html.md b/website/source/guides/tls.html.md index 7135f94be63..98f1a8e0726 100644 --- a/website/source/guides/tls.html.md +++ b/website/source/guides/tls.html.md @@ -222,17 +222,20 @@ the node. This guide assumes they're in Nomad's current directory. ``` These two settings are important for ensuring all of Nomad's mTLS security -properties are met. `verify_server_hostname` may be set to `false` to only -ensure that a node's certificate is signed by the same CA. This means any -service with a certificate from the same CA as Nomad can act as a client or -server of any region. +properties are met. If `verify_server_hostname` is set to `false` the node's +cerificate will be checked to ensure it is signed by the same CA, but its role +and region will not be verified. This means any service with a certificate from +the same CA as Nomad can act as a client or server of any region. -`verify_https_client` may be disabled to allow non-Nomad clients (eg Consul or -curl) to communicate with the HTTPS API. +`verify_https_client` may be disabled to allow HTTP API clients (eg Nomad CLI, Consul, or +curl) to communicate with the HTTPS API without presenting a client-side +certificate. If `verify_https_client` is enabled ony HTTP API clients +presenting a certificate signed by the same CA as Nomad's certificate are +allowed to access Nomad. ~> Enabling `verify_https_client` feature effectively protects Nomad from unauthorized network access at the cost of breaking compatibility with Consul - HTTPS health checks and third party tools like curl. + HTTPS health checks. ### Client configuration From c2dc7f4677e5bdd662fe126e860e2a83cb2df9da Mon Sep 17 00:00:00 2001 From: Michael Schurter Date: Thu, 27 Jul 2017 14:47:43 -0700 Subject: [PATCH 03/13] Switch to cfssl and get everything working --- demo/vagrant/Vagrantfile | 10 +- website/source/guides/tls.html.md | 223 +++++++++++++++--------------- 2 files changed, 116 insertions(+), 117 deletions(-) diff --git a/demo/vagrant/Vagrantfile b/demo/vagrant/Vagrantfile index 1742b1ee14e..34d0448f631 100644 --- a/demo/vagrant/Vagrantfile +++ b/demo/vagrant/Vagrantfile @@ -7,8 +7,7 @@ sudo apt-get update sudo DEBIAN_FRONTEND=noninteractive apt-get install -y unzip curl vim \ apt-transport-https \ ca-certificates \ - software-properties-common \ - openssl + software-properties-common # Download Nomad NOMAD_VERSION=0.6.0 @@ -68,6 +67,13 @@ EOF sudo systemctl enable consul.service sudo systemctl start consul +for bin in cfssl cfssl-certinfo cfssljson +do + echo "Installing $bin..." + curl -sSL https://pkg.cfssl.org/R1.2/${bin}_linux-amd64 > /tmp/${bin} + sudo install /tmp/${bin} /usr/local/bin/${bin} +done + SCRIPT Vagrant.configure(2) do |config| diff --git a/website/source/guides/tls.html.md b/website/source/guides/tls.html.md index 98f1a8e0726..ba325e735de 100644 --- a/website/source/guides/tls.html.md +++ b/website/source/guides/tls.html.md @@ -24,7 +24,7 @@ the expected region and configured for the expected role (e.g. `client.us-west.nomad`). Configuring TLS can be unfortunately complex process, but if you used the -[Getting Started guide's Vagrantfile][Vagrantfile] or have [OpenSSL][] and Nomad +[Getting Started guide's Vagrantfile][Vagrantfile] or have [cfssl][] and Nomad installed this guide will provide you with a production ready TLS configuration. @@ -38,72 +38,75 @@ XXX TODO XXX - serf encryption key The first step to configuring TLS for Nomad is generating certificates. In order to prevent unauthorized cluster access, Nomad requires all certificates -are signed by the sign Certificate Authority (CA). This should be a *private* -CA and not a public like [Let's Encrypt][letsencrypt] as any certificate signed -by this CA will be allowed to communicate with the cluster. +be signed by the same Certificate Authority (CA). This should be a *private* CA +and not a public one like [Let's Encrypt][letsencrypt] as any certificate +signed by this CA will be allowed to communicate with the cluster. ### Certificate Authority -You can generate a private CA certificate and key with OpenSSL: +There are a variety of tools for managing your own CA, [like the PKI secret +backend in Vault][vault-pki], but for the sake of simplicity in this guide +we'll use [cfssl][]. You can generate a private CA certificate and key with +[cfssl][]: ```shell -# Generate the CA's private key -# This file (nomad-ca.key) must be kept *secret* -openssl genrsa -out nomad-ca.key 4096 - -# Generate the CA's self-signed certicate -# This file (nomad-ca.crt) will be distributed to all nodes -openssl req -new -x509 -key nomad-ca.key -out nomad-ca.crt -You are about to be asked to enter information that will be incorporated -into your certificate request. -What you are about to enter is what is called a Distinguished Name or a DN. -There are quite a few fields but you can leave some blank -For some fields there will be a default value, -If you enter '.', the field will be left blank. ------ -Country Name (2 letter code) [AU]:. -State or Province Name (full name) [Some-State]:. -Locality Name (eg, city) []:. -Organization Name (eg, company) [Internet Widgits Pty Ltd]:. -Organizational Unit Name (eg, section) []:. -Common Name (e.g. server FQDN or YOUR name) []:Nomad CA -Email Address []:. +# Generate the CA's private key and certificate +cfssl print-defaults csr | cfssl gencert -initca - | cfssljson -bare nomad-ca ``` -Your answers to OpenSSL's prompts are purely informational and not used by -Nomad. - -The CA key (`nomad-ca.key`) will be used to sign certificates for Nomad nodes -and must be kept private. The CA certificate (`nomad-ca.crt`) contains the -public key necessary to validate Nomad certificates and therefore must be +The CA key (`nomad-ca-key.pem`) will be used to sign certificates for Nomad +nodes and must be kept private. The CA certificate (`nomad-ca.pem`) contains +the public key necessary to validate Nomad certificates and therefore must be distributed to every node that requires access. ### Node Certificates Once you have a CA certifacte and key you can generate and sign the -certificates Nomad will use directly. Traditionally TLS certificates use the +certificates Nomad will use directly. TLS certificates commonly use the fully-qualified domain name of the system being identified as the certificate's Common Name (CN). However, hosts (and therefore hostnames and IPs) are often ephemeral in Nomad clusters. They come and go as clusters are scaled up and down or outages occur. Not only would signing a new certificate per Nomad node be difficult, but using a hostname provides no security or functional benefits -to Nomad. To fulfill the desired security properties (see above) Nomad -certificates are signed with their region and role such as: +to Nomad. To fulfill the desired security properties (above) Nomad certificates +are signed with their region and role such as: * `client.global.nomad` for a client node in the `global` region * `server.us-west.nomad` for a server node in the `us-west` region To create certificates for the client and server in the cluster from the -[Getting Started guide][guide-cluster] with OpenSSL create the following -configuration file `nomad.conf`: +[Getting Started guide][guide-cluster] with [cfssl][] create ([or +download][cfssl.json]) the following configuration file as `cfssl.json` to +increase the default certificate expiration time: + +```json +{ + "signing": { + "default": { + "expiry": "87600h", + "usages": [ + "signing", + "key encipherment", + "server auth", + "client auth" + ] + } + } +} +``` -```ini -basicConstraints = CA:FALSE -subjectAltName = @alt_names +```shell +# Generate a certificate for the Nomad server +echo '{}' | cfssl gencert -ca=nomad-ca.pem -ca-key=nomad-ca-key.pem -config=cfssl.json \ + -hostname="server.global.nomad,localhost" - | cfssljson -bare server -[alt_names] -DNS.1 = ${commonName} -DNS.2 = localhost +# Generate a certificate for the Nomad client +echo '{}' | cfssl gencert -ca=nomad-ca.pem -ca-key=nomad-ca-key.pem -config=cfssl.json \ + -hostname="client.global.nomad,localhost" - | cfssljson -bare client + +# Generate a certificate for the CLI +echo '{}' | cfssl gencert -ca nomad-ca.pem -ca-key nomad-ca-key.pem -profile=client \ + - | cfssljson -bare cli ``` Using `localhost` as a subject alternate name (SAN) allows tools like `curl` to @@ -111,61 +114,32 @@ be able to communicate with Nomad's HTTP API when run on the same host. Other SANs may be added including a DNS resolvable hostname to allow remote HTTP requests from third party tools. -Then create client and server certificate and key pairs: - -```shell -# Client key and certificate -openssl genrsa -out client.global.nomad.key 4096 -openssl req -new -sha256 \ - -out client.global.nomad.csr \ - -key client.global.nomad.key \ - -subj /CN=client.global.nomad/ -openssl x509 -req \ - -in client.global.nomad.csr \ - -CA nomad-ca.crt \ - -CAkey nomad-ca.key \ - -days 3650 \ - -set_serial $(hexdump -e '"0x%x%x%x%x"' -n 16 /dev/urandom) \ - -extfile nomad.conf \ - -out client.global.nomad.crt - -# Server key and certificate -openssl genrsa -out server.global.nomad.key 4096 -openssl req -new -sha256 \ - -out server.global.nomad.csr \ - -key server.global.nomad.key \ - -subj /CN=server.global.nomad/ -openssl x509 -req \ - -in server.global.nomad.csr \ - -CA nomad-ca.crt \ - -CAkey nomad-ca.key \ - -days 3650 \ - -set_serial $(hexdump -e '"0x%x%x%x%x"' -n 16 /dev/urandom) \ - -extfile nomad.conf \ - -out server.global.nomad.crt -``` - You should now have the following files: -* `nomad-ca.key` - CA private key. Keep safe! -* `nomad-ca.crt` - CA public certificate. -* `client.global.nomad.key` - Nomad client node private key for the `global` region. -* `client.global.nomad.csr` - Nomad client node certificate signing request for the `global` region. -* `client.global.nomad.crt` - Nomad client node public certificate for the `global` region. -* `server.global.nomad.key` - Nomad server node private key for the `global` region. -* `server.global.nomad.csr` - Nomad server node certificate signing request for the `global` region. -* `server.global.nomad.crt` - Nomad server node public certificate for the `global` region. - -Each Nomad node should have the appropriate key (`.key`) and certificate -(`.crt`) file for its region and role. In addition each node needs the CA's -public certificate (`nomad-ca.crt`). +* `cfssl.json` - cfssl configuration. +* `nomad-ca.csr` - CA signing request. +* `nomad-ca-key.pem` - CA private key. Keep safe! +* `nomad-ca.pem` - CA public certificate. +* `cli.csr` - Nomad CLI certificate signing request. +* `cli.pem` - Nomad CLI certificate. +* `cli-key.pem` - Nomad CLI private key. +* `client.csr` - Nomad client node certificate signing request for the `global` region. +* `client-key.pem` - Nomad client node private key for the `global` region. +* `client.pem` - Nomad client node public certificate for the `global` region. +* `server.csr` - Nomad server node certificate signing request for the `global` region. +* `server-key.pem` - Nomad server node private key for the `global` region. +* `server.pem` - Nomad server node public certificate for the `global` region. + +Each Nomad node should have the appropriate key (`-key.pem`) and certificate +(`.pem`) file for its region and role. In addition each node needs the CA's +public certificate (`nomad-ca.pem`). ## Configuring Nomad Once you have the appropriate key and certificates installed you're ready to configure Nomad to use them for mTLS. Starting with the [server configuration -from the Getting Started guide][guide-server] add the following TLS specific -configuration options: +from the Getting Started guide][guide-server] add the following TLS +CONFIGUration options: ```hcl # Increase log verbosity @@ -187,9 +161,9 @@ tls { http = true rpc = true - ca_file = "nomad-ca.crt" - cert_file = "server.global.nomad.crt" - key_file = "server.global.nomad.key" + ca_file = "nomad-ca.pem" + cert_file = "server.pem" + key_file = "server-key.pem" verify_server_hostname = true verify_https_client = true @@ -208,13 +182,13 @@ doesn't use separate ports for TLS and non-TLS traffic: your cluster should either use TLS or not. ```hcl - ca_file = "nomad-ca.crt" - cert_file = "server.global.nomad.crt" - key_file = "server.global.nomad.key" + ca_file = "nomad-ca.pem" + cert_file = "server.pem" + key_file = "server-key.pem" ``` The file lines should point to whereever you placed the certificate files on -the node. This guide assumes they're in Nomad's current directory. +the node. This guide assumes they are in Nomad's current directory. ```hcl verify_server_hostname = true @@ -227,11 +201,12 @@ cerificate will be checked to ensure it is signed by the same CA, but its role and region will not be verified. This means any service with a certificate from the same CA as Nomad can act as a client or server of any region. -`verify_https_client` may be disabled to allow HTTP API clients (eg Nomad CLI, Consul, or -curl) to communicate with the HTTPS API without presenting a client-side -certificate. If `verify_https_client` is enabled ony HTTP API clients -presenting a certificate signed by the same CA as Nomad's certificate are -allowed to access Nomad. +`verify_https_client` requires HTTP API clients to present a certificate signed +by the same CA as Nomad's certificate. It may be disabled to allow HTTP API +clients (eg Nomad CLI, Consul, or curl) to communicate with the HTTPS API +without presenting a client-side certificate. If `verify_https_client` is +enabled ony HTTP API clients presenting a certificate signed by the same CA as +Nomad's certificate are allowed to access Nomad. ~> Enabling `verify_https_client` feature effectively protects Nomad from unauthorized network access at the cost of breaking compatibility with Consul @@ -269,9 +244,9 @@ tls { http = true rpc = true - ca_file = "nomad-ca.crt" - cert_file = "client.global.nomad.crt" - key_file = "client.global.nomad.key" + ca_file = "nomad-ca.pem" + cert_file = "client.pem" + key_file = "client-key.pem" verify_server_hostname = true verify_https_client = true @@ -307,17 +282,33 @@ Don't worry, the Nomad CLI just defaults to `http://...` instead of ```shell export NOMAD_ADDR=https://localhost:4646 -export NOMAD_CACERT=nomad-ca.crt -export NOMAD_CLIENT_CERT=client.global.nomad.crt +export NOMAD_CACERT=nomad-ca.pem +export NOMAD_CLIENT_CERT=client.pem +export NOMAD_CLIENT_KEY=client-key.pem ``` The `NOMAD_CACERT` also needs to be set so the CLI can verify it's talking to -an actual Nomad node. Finally, the `NOMAD_CLIENT_CERT` needs to be set since we -enabled `verify_https_client` above which prevents any access lacking a client -certificate. Operators may wish to generate a certificate specifically for the -CLI as any certificate signed by Nomad's CA will work. +an actual Nomad node. Finally, `NOMAD_CLIENT_CERT` and `NOMAD_CLIENT_KEY` need +to be set since we enabled `verify_https_client` above which prevents any +access lacking a client certificate. + +Now the CLI works as expected: -XXX TODO XXX - an example of everything working +```text +vagrant@nomad:~$ nomad node-status +ID DC Name Class Drain Status +237cd4c5 dc1 nomad false ready + +vagrant@nomad:~$ nomad init +Example job file written to example.nomad +vagrant@nomad:~$ nomad run example.nomad +==> Monitoring evaluation "e9970e1d" + Evaluation triggered by job "example" + Allocation "a1f6c3e7" created: node "237cd4c5", group "cache" + Evaluation within deployment: "080460ce" + Evaluation status changed: "pending" -> "complete" +==> Evaluation "e9970e1d" finished with status "complete" +``` ## Switching an existing cluster to TLS @@ -326,6 +317,8 @@ XXX TODO XXX [guide-server]: https://raw.githubusercontent.com/hashicorp/nomad/master/demo/vagrant/server.hcl [guide-cluster]: https://www.nomadproject.io/intro/getting-started/cluster.html [letsencrypt]: https://letsencrypt.org/ -[OpenSSL]: https://www.openssl.org/ +[cfssl]: https://cfssl.org/ +[cfssl.json]: https://raw.githubusercontent.com/hashicorp/nomad/master/demo/vagrant/cfssl.json [Vagrantfile]: https://raw.githubusercontent.com/hashicorp/nomad/master/demo/vagrant/Vagrantfile -[Vault]: https://www.vaultproject.io/docs/secrets/pki/index.html +[Vault]: https://www.vaultproject.io/ +[vault-pki]: https://www.vaultproject.io/docs/secrets/pki/index.html From 1676f101e8e5d9f40edbfb33062f061ecdba6a1f Mon Sep 17 00:00:00 2001 From: Michael Schurter Date: Thu, 27 Jul 2017 15:32:02 -0700 Subject: [PATCH 04/13] Add serf section and clean it up --- website/source/guides/tls.html.md | 59 ++++++++++++++++++++++++++++--- 1 file changed, 55 insertions(+), 4 deletions(-) diff --git a/website/source/guides/tls.html.md b/website/source/guides/tls.html.md index ba325e735de..1c71c38dbc1 100644 --- a/website/source/guides/tls.html.md +++ b/website/source/guides/tls.html.md @@ -3,7 +3,9 @@ layout: "guides" page_title: "Securing Nomad with TLS" sidebar_current: "guides-tls" description: |- - Securing Nomad's cluster communication with TLS is XXX TODO XXX + Securing Nomad's cluster communication is not only important for security but + can even ease operations by preventing mistakes and misconfigurations. Nomad + optionally uses mutual TLS (mTLS) for all HTTP and RPC communication. --- # Securing Nomad with TLS @@ -32,8 +34,6 @@ configuration. management and rotation is a complex subject not covered by this guide. [Vault][] is the suggested solution for key generation and management. -XXX TODO XXX - serf encryption key - ## Creating Certificates The first step to configuring TLS for Nomad is generating certificates. In @@ -310,9 +310,60 @@ vagrant@nomad:~$ nomad run example.nomad ==> Evaluation "e9970e1d" finished with status "complete" ``` +## Server Gossip + +We haven't quite completely secured Nomad's communications: Nomad server's +gossip protocol uses a shared key instead of TLS for encryption. This +encryption key must be added to every server's configuration using the +[`encrypt`](/docs/agent/configuration/server.html#encrypt) parameter. + +As a convenience the Nomad CLI includes a `keygen` command for generating a new +secure gossip encryption key: + +```text +$ nomad keygen +cg8StVXbQJ0gPvMd9o7yrg== +``` + +Put the generated key into each server's configuration file: + +```hcl +server { + enabled = true + + # Self-elect, should be 3 or 5 for production + bootstrap_expect = 1 + + # Encrypt gossip communication + encrypt = "cg8StVXbQJ0gPvMd9o7yrg==" +} +``` + ## Switching an existing cluster to TLS -XXX TODO XXX +Since Nomad does *not* use different ports for TLS and non-TLS communication, +the use of TLS should be consistent across the cluster. Switching an existing +cluster to use TLS everywhere is similar to upgrading between versions of +Nomad. + +First make sure all of your nodes are ready to be switched: + +* Add the appropriate key and certificates to all nodes. + * Ensure the private key file is only readable by the Nomad user. +* Add the environment variables to all nodes where the CLI is used. +* Add the appropriate `tls` block to the configuration file on all nodes. +* Generate a gossip key and add it the Nomad server configuration. + +At this point a rolling restart of the cluster will enable TLS everywhere. + +1. Restart servers, one at a time +2. Restart clients, one or more at a time + +~> As soon as a quorum of servers are TLS-enabled, clients will not be able to + communicate with them until they are restarted. + +Jobs running in the cluster will *not* be affected and will continue running +throughout the switch. [guide-server]: https://raw.githubusercontent.com/hashicorp/nomad/master/demo/vagrant/server.hcl [guide-cluster]: https://www.nomadproject.io/intro/getting-started/cluster.html From cb2b3ff2fcf77a0b9a27432071be0bbf7de89059 Mon Sep 17 00:00:00 2001 From: Michael Schurter Date: Thu, 27 Jul 2017 16:03:38 -0700 Subject: [PATCH 05/13] Remove old example tls config It didn't work in cfssl 1.2 anyway (required building from cfssl master). --- website/source/docs/agent/encryption.html.md | 89 ++------------------ website/source/guides/tls.html.md | 2 +- website/source/layouts/guides.erb | 4 + 3 files changed, 10 insertions(+), 85 deletions(-) diff --git a/website/source/docs/agent/encryption.html.md b/website/source/docs/agent/encryption.html.md index ed381ccc5ff..aa0446745ec 100644 --- a/website/source/docs/agent/encryption.html.md +++ b/website/source/docs/agent/encryption.html.md @@ -34,7 +34,7 @@ With that key, you can enable gossip encryption on the agent. Nomad supports using TLS to verify the authenticity of servers and clients. To enable this, Nomad requires that all clients and servers have key pairs that are -generated and signed by a Certificate Authority. This can be a private CA. +generated and signed by a private Certificate Authority (CA). TLS can be used to verify the authenticity of the servers and clients. The configuration option [`verify_server_hostname`][tls] causes Nomad to verify that @@ -82,89 +82,10 @@ connections as well as verify proper names on all other certificates. Consul will not attempt to health check agents with `verify_https_client` set as it is unable to use client certificates. -## Encryption Examples +# Configuring Nomad with TLS -### TLS Configuration using `cfssl` +Read the [Configuring TLS Guide][guide] for details on how to configure +encryption for Nomad. -While [Vault's PKI backend][vault] is an ideal solution for managing -certificates and other secrets in a production environment, it's useful to use -simpler command line tools when learning how to configure TLS and your [PKI]. - -[`cfssl`][cfssl] is a tool for working with TLS certificates and certificate -authorities similar to [OpenSSL's][openssl] `x509` command line tool. - -Once you have the `cfssl` command line tool installed, the first step to -setting up TLS is to create a Certificate Authority (CA) certificate. The -following command will generate a suitable example CA CSR, certificate, and -key: - -```shell -# Run in the directory where you want to store certificates -$ cfssl print-defaults csr | cfssl gencert -initca - | cfssljson -bare ca -``` - -Next create a `nomad-csr.json` which contains the configuration for the actual -certificate you'll be using in Nomad: - -```json -{ - "CN": "global.nomad", - "hosts": [ - "server.global.nomad", - "client.global.nomad", - "localhost" - ] -} -``` - -This will create a certificate suitable for both clients and servers in the -`global` (default) region. - -In production Nomad agents should have a certificate valid for the name -`${ROLE}.${REGION}.nomad` where role is either `client` or `server` depending -on the node's role. - -Create a certificate signed by your CA: - -```shell -$ cfssl gencert -ca ca.pem -ca-key ca-key.pem nomad-csr.json | cfssljson -bare nomad -``` - -You've now successfully generated self-signed certificates! You should see the -following files: - -- `ca.pem` - the CA certificate. This corresponds to the Nomad `ca_file` - parameter in the Nomad [`tls` stanza][tls]. - -- `ca-key.pem` - the CA private key. This is used to sign CSRs and should - **not** be included in the Nomad [`tls` stanza][tls]. - -- `nomad.pem` - the Nomad certificate for the region. This corresponds to the - `cert_file` parameter in the Nomad [`tls` stanza][tls]. - -- `nomad-key.pem` - the Nomad private key. This corresponds to the `key_file` - parameter in the Nomad [`tls` stanza][tls]. - -- `*.csr` - the certificate signing request. This is temporary for generating - certificates and should **not** be included in the Nomad [`tls` stanza][tls]. - -In your Nomad configuration add the `tls` stanza: - -```hcl -tls { - http = true - rpc = true - - ca_file = "ca.pem" - cert_file = "nomad.pem" - key_file = "nomad-key.pem" - - verify_server_hostname = true -} -``` - -[vault]: https://www.vaultproject.io/docs/secrets/pki/ -[PKI]: https://en.wikipedia.org/wiki/Public_key_infrastructure -[cfssl]: https://cfssl.org/ -[openssl]: https://www.openssl.org/ +[guide]: /guides/tls.html "Configuring TLS" [tls]: /docs/agent/configuration/tls.html "Nomad TLS Configuration" diff --git a/website/source/guides/tls.html.md b/website/source/guides/tls.html.md index 1c71c38dbc1..ab8f54404f3 100644 --- a/website/source/guides/tls.html.md +++ b/website/source/guides/tls.html.md @@ -26,7 +26,7 @@ the expected region and configured for the expected role (e.g. `client.us-west.nomad`). Configuring TLS can be unfortunately complex process, but if you used the -[Getting Started guide's Vagrantfile][Vagrantfile] or have [cfssl][] and Nomad +[Getting Started Guide's Vagrantfile][Vagrantfile] or have [cfssl][] and Nomad installed this guide will provide you with a production ready TLS configuration. diff --git a/website/source/layouts/guides.erb b/website/source/layouts/guides.erb index b6089ad59d3..c844c058349 100644 --- a/website/source/layouts/guides.erb +++ b/website/source/layouts/guides.erb @@ -50,6 +50,10 @@ + > + Configuring TLS + + > Outage Recovery From 4fa43440b10bc203caed06aeebf3cfb4f8733e77 Mon Sep 17 00:00:00 2001 From: Michael Schurter Date: Tue, 1 Aug 2017 15:40:36 -0700 Subject: [PATCH 06/13] Lots of fixes from @sethvargo * hclfmt the world * 2 space indent * make every example well formed with stanzas and comments * jsonfmt too * mdfmt manually * _ instead of * * no [links][], only [links][links] * ordered lists instead of bullets when possible * lots of wording fixes * de-contractionization * add 127.0.0.1 to SANs * -1 on intentional errors * -1 on first person --- demo/vagrant/client1.hcl | 12 +- demo/vagrant/client2.hcl | 12 +- demo/vagrant/server.hcl | 6 +- dist/client.hcl | 6 +- dist/server.hcl | 12 +- .../source/docs/commands/agent.html.md.erb | 2 + website/source/guides/tls.html.md | 247 +++++++++++------- 7 files changed, 175 insertions(+), 122 deletions(-) diff --git a/demo/vagrant/client1.hcl b/demo/vagrant/client1.hcl index 53bc4e94ae2..845a4ebe629 100644 --- a/demo/vagrant/client1.hcl +++ b/demo/vagrant/client1.hcl @@ -6,15 +6,15 @@ data_dir = "/tmp/client1" # Enable the client client { - enabled = true + enabled = true - # For demo assume we are talking to server1. For production, - # this should be like "nomad.service.consul:4647" and a system - # like Consul used for service discovery. - servers = ["127.0.0.1:4647"] + # For demo assume we are talking to server1. For production, + # this should be like "nomad.service.consul:4647" and a system + # like Consul used for service discovery. + servers = ["127.0.0.1:4647"] } # Modify our port to avoid a collision with server1 ports { - http = 5656 + http = 5656 } diff --git a/demo/vagrant/client2.hcl b/demo/vagrant/client2.hcl index 44dcf2a822f..963546c4bab 100644 --- a/demo/vagrant/client2.hcl +++ b/demo/vagrant/client2.hcl @@ -6,15 +6,15 @@ data_dir = "/tmp/client2" # Enable the client client { - enabled = true + enabled = true - # For demo assume we are talking to server1. For production, - # this should be like "nomad.service.consul:4647" and a system - # like Consul used for service discovery. - servers = ["127.0.0.1:4647"] + # For demo assume we are talking to server1. For production, + # this should be like "nomad.service.consul:4647" and a system + # like Consul used for service discovery. + servers = ["127.0.0.1:4647"] } # Modify our port to avoid a collision with server1 and client1 ports { - http = 5657 + http = 5657 } diff --git a/demo/vagrant/server.hcl b/demo/vagrant/server.hcl index 653b2c0379e..f055de4e6f6 100644 --- a/demo/vagrant/server.hcl +++ b/demo/vagrant/server.hcl @@ -6,8 +6,8 @@ data_dir = "/tmp/server1" # Enable the server server { - enabled = true + enabled = true - # Self-elect, should be 3 or 5 for production - bootstrap_expect = 1 + # Self-elect, should be 3 or 5 for production + bootstrap_expect = 1 } diff --git a/dist/client.hcl b/dist/client.hcl index 774f3ac2a03..c0848282eac 100644 --- a/dist/client.hcl +++ b/dist/client.hcl @@ -2,6 +2,6 @@ bind_addr = "127.0.0.1" data_dir = "/var/lib/nomad/" client { - enabled = true - servers = ["10.1.0.1", "10.1.0.2", "10.1.0.3"] -} \ No newline at end of file + enabled = true + servers = ["10.1.0.1", "10.1.0.2", "10.1.0.3"] +} diff --git a/dist/server.hcl b/dist/server.hcl index 82d9673b12b..0ce33aeb6e4 100644 --- a/dist/server.hcl +++ b/dist/server.hcl @@ -2,12 +2,12 @@ bind_addr = "0.0.0.0" data_dir = "/var/lib/nomad" advertise { - # This should be the IP of THIS MACHINE and must be routable by every node - # in your cluster - rpc = "1.2.3.4:4647" + # This should be the IP of THIS MACHINE and must be routable by every node + # in your cluster + rpc = "1.2.3.4:4647" } server { - enabled = true - bootstrap_expect = 3 -} \ No newline at end of file + enabled = true + bootstrap_expect = 3 +} diff --git a/website/source/docs/commands/agent.html.md.erb b/website/source/docs/commands/agent.html.md.erb index 3544a8c506b..86d06c9c366 100644 --- a/website/source/docs/commands/agent.html.md.erb +++ b/website/source/docs/commands/agent.html.md.erb @@ -35,6 +35,8 @@ via CLI arguments. The `agent` command accepts the following arguments: * `-dev`: Start the agent in development mode. This enables a pre-configured dual-role agent (client + server) which is useful for developing or testing Nomad. No other configuration is required to start the agent in this mode. +* `-encrypt`: Set the Serf encryption key. See [Agent + Encryption](/docs/agent/encryption.html) for more details. * `-join=
`: Address of another agent to join upon starting up. This can be specified multiple times to specify multiple agents to join. * `-log-level=`: Equivalent to the [log_level](#log_level) config option. diff --git a/website/source/guides/tls.html.md b/website/source/guides/tls.html.md index ab8f54404f3..14ef2fcf039 100644 --- a/website/source/guides/tls.html.md +++ b/website/source/guides/tls.html.md @@ -3,51 +3,61 @@ layout: "guides" page_title: "Securing Nomad with TLS" sidebar_current: "guides-tls" description: |- - Securing Nomad's cluster communication is not only important for security but - can even ease operations by preventing mistakes and misconfigurations. Nomad - optionally uses mutual TLS (mTLS) for all HTTP and RPC communication. + Securing Nomad's cluster communication with TLS is important for both + security and easing operations. Nomad can use mutual TLS (mTLS) for + authenticating for all HTTP and RPC communication. --- # Securing Nomad with TLS Securing Nomad's cluster communication is not only important for security but can even ease operations by preventing mistakes and misconfigurations. Nomad -optionally uses mutual TLS (mTLS) for all HTTP and RPC communication. Nomad's -use of mTLS provides the following properties: +optionally uses mutual [TLS][tls] (mTLS) for all HTTP and RPC communication. +Nomad's use of mTLS provides the following properties: * Prevent unauthorized Nomad access * Prevent observing or tampering with Nomad communication * Prevent client/server role or region misconfigurations +* Prevent other services from masquerading as Nomad agents -The 3rd property is fairly unique to Nomad's use of TLS. While most uses of TLS -verify the identity of the server you're connecting to based on a domain name -such as `nomadproject.io`, Nomad verifies the node you're connecting to is in +Preventing region misconfigurations is a property of Nomad's mTLS not commonly +found in the TLS implementations on the public Internet. While most uses of +TLS verify the identity of the server you are connecting to based on a domain +name such as `example.com`, Nomad verifies the node you are connecting to is in the expected region and configured for the expected role (e.g. -`client.us-west.nomad`). +`client.us-west.nomad`). This also prevents other services who may have access +to certificates signed by the same private CA from masquerading as Nomad +agents. If certificates were identified based on hostname/IP then any other +service on a host could masquerade as a Nomad agent. -Configuring TLS can be unfortunately complex process, but if you used the -[Getting Started Guide's Vagrantfile][Vagrantfile] or have [cfssl][] and Nomad -installed this guide will provide you with a production ready TLS -configuration. +Correctly configuring TLS can be a complex process, especially given the wide +range of deployment methodologies. If you use the sample +[Vagrantfile][vagrantfile] from the [Getting Started Guide][guide-install] - or +have [cfssl][cfssl] and Nomad installed - this guide will provide you with a +production ready TLS configuration. ~> Note that while Nomad's TLS configuration will be production ready, key management and rotation is a complex subject not covered by this guide. - [Vault][] is the suggested solution for key generation and management. + [Vault][vault] is the suggested solution for key generation and management. ## Creating Certificates The first step to configuring TLS for Nomad is generating certificates. In order to prevent unauthorized cluster access, Nomad requires all certificates -be signed by the same Certificate Authority (CA). This should be a *private* CA +be signed by the same Certificate Authority (CA). This should be a _private_ CA and not a public one like [Let's Encrypt][letsencrypt] as any certificate signed by this CA will be allowed to communicate with the cluster. +~> Nomad certificates may be signed by different intermediate CAs as long as + the full `ca_file` on each node contains all of the CA certificates in the + chain. + ### Certificate Authority There are a variety of tools for managing your own CA, [like the PKI secret -backend in Vault][vault-pki], but for the sake of simplicity in this guide -we'll use [cfssl][]. You can generate a private CA certificate and key with -[cfssl][]: +backend in Vault][vault-pki], but for the sake of simplicity this guide will +use [cfssl][cfssl]. You can generate a private CA certificate and key with +[cfssl][cfssl]: ```shell # Generate the CA's private key and certificate @@ -65,44 +75,43 @@ Once you have a CA certifacte and key you can generate and sign the certificates Nomad will use directly. TLS certificates commonly use the fully-qualified domain name of the system being identified as the certificate's Common Name (CN). However, hosts (and therefore hostnames and IPs) are often -ephemeral in Nomad clusters. They come and go as clusters are scaled up and -down or outages occur. Not only would signing a new certificate per Nomad node -be difficult, but using a hostname provides no security or functional benefits -to Nomad. To fulfill the desired security properties (above) Nomad certificates -are signed with their region and role such as: +ephemeral in Nomad clusters. Not only would signing a new certificate per +Nomad node be difficult, but using a hostname provides no security or +functional benefits to Nomad. To fulfill the desired security properties +(above) Nomad certificates are signed with their region and role such as: * `client.global.nomad` for a client node in the `global` region * `server.us-west.nomad` for a server node in the `us-west` region To create certificates for the client and server in the cluster from the -[Getting Started guide][guide-cluster] with [cfssl][] create ([or +[Getting Started guide][guide-cluster] with [cfssl][cfssl] create ([or download][cfssl.json]) the following configuration file as `cfssl.json` to increase the default certificate expiration time: ```json { - "signing": { - "default": { - "expiry": "87600h", - "usages": [ - "signing", - "key encipherment", - "server auth", - "client auth" - ] - } + "signing": { + "default": { + "expiry": "87600h", + "usages": [ + "signing", + "key encipherment", + "server auth", + "client auth" + ] } + } } ``` ```shell # Generate a certificate for the Nomad server echo '{}' | cfssl gencert -ca=nomad-ca.pem -ca-key=nomad-ca-key.pem -config=cfssl.json \ - -hostname="server.global.nomad,localhost" - | cfssljson -bare server + -hostname="server.global.nomad,localhost,127.0.0.1" - | cfssljson -bare server # Generate a certificate for the Nomad client echo '{}' | cfssl gencert -ca=nomad-ca.pem -ca-key=nomad-ca-key.pem -config=cfssl.json \ - -hostname="client.global.nomad,localhost" - | cfssljson -bare client + -hostname="client.global.nomad,localhost,127.0.0.1" - | cfssljson -bare client # Generate a certificate for the CLI echo '{}' | cfssl gencert -ca nomad-ca.pem -ca-key nomad-ca-key.pem -profile=client \ @@ -136,10 +145,10 @@ public certificate (`nomad-ca.pem`). ## Configuring Nomad -Once you have the appropriate key and certificates installed you're ready to -configure Nomad to use them for mTLS. Starting with the [server configuration -from the Getting Started guide][guide-server] add the following TLS -CONFIGUration options: +Next Nomad must be configured to use the newly-created key and certificates for +mTLS. Starting with the [server configuration from the Getting Started +guide][guide-server] add the following TLS +configuration options: ```hcl # Increase log verbosity @@ -150,10 +159,10 @@ data_dir = "/tmp/server1" # Enable the server server { - enabled = true + enabled = true - # Self-elect, should be 3 or 5 for production - bootstrap_expect = 1 + # Self-elect, should be 3 or 5 for production + bootstrap_expect = 1 } # Require TLS @@ -173,8 +182,11 @@ tls { The new `tls` section is worth breaking down in more detail: ```hcl +tls { http = true rpc = true + # ... +} ``` This enables TLS for the HTTP and RPC protocols. Unlike web servers, Nomad @@ -182,24 +194,34 @@ doesn't use separate ports for TLS and non-TLS traffic: your cluster should either use TLS or not. ```hcl +tls { + # ... + ca_file = "nomad-ca.pem" cert_file = "server.pem" key_file = "server-key.pem" + + # ... +} ``` The file lines should point to whereever you placed the certificate files on the node. This guide assumes they are in Nomad's current directory. ```hcl +tls { + # ... + verify_server_hostname = true verify_https_client = true +} ``` These two settings are important for ensuring all of Nomad's mTLS security properties are met. If `verify_server_hostname` is set to `false` the node's cerificate will be checked to ensure it is signed by the same CA, but its role -and region will not be verified. This means any service with a certificate from -the same CA as Nomad can act as a client or server of any region. +and region will not be verified. This means any service with a certificate +signed by same CA as Nomad can act as a client or server of any region. `verify_https_client` requires HTTP API clients to present a certificate signed by the same CA as Nomad's certificate. It may be disabled to allow HTTP API @@ -212,10 +234,10 @@ Nomad's certificate are allowed to access Nomad. unauthorized network access at the cost of breaking compatibility with Consul HTTPS health checks. -### Client configuration +### Client Configuration -The Nomad client configuration is similar with the only difference being the -certificate and key used: +The Nomad client configuration is similar to the server configuration. The +biggest difference is in the certificate and key used for configuration. ```hcl # Increase log verbosity @@ -226,17 +248,17 @@ data_dir = "/tmp/client1" # Enable the client client { - enabled = true + enabled = true - # For demo assume we are talking to server1. For production, - # this should be like "nomad.service.consul:4647" and a system - # like Consul used for service discovery. - servers = ["127.0.0.1:4647"] + # For demo assume we are talking to server1. For production, + # this should be like "nomad.service.consul:4647" and a system + # like Consul used for service discovery. + servers = ["127.0.0.1:4647"] } # Modify our port to avoid a collision with server1 ports { - http = 5656 + http = 5656 } # Require TLS @@ -268,38 +290,48 @@ nomad agent -config server1.hcl nomad agent -config client1.hcl ``` -Finally in a third terminal test out `nomad node-status`: +If you run `nomad node-status` now, you'll get an error, like: -```text -vagrant@nomad:~$ nomad node-status +``` Error querying node status: Get http://127.0.0.1:4646/v1/nodes: malformed HTTP response "\x15\x03\x01\x00\x02\x02" ``` -Oh no! That didn't work! +This is because the Nomad CLI defaults to communicating via HTTP instead of +HTTPS. We can configure the local Nomad client to connect using TLS and specify +our custom keys and certificates using the command line: + +```shell +nomad node-status -ca-cert=nomad-ca.pem -client-cert=cli.pem -client-key=cli-key.pem -addr=https://127.0.0.1:4646 +``` -Don't worry, the Nomad CLI just defaults to `http://...` instead of -`https://...`. We can override this with an environment variable: +This process can be cumbersome to type each time, so the Nomad CLI also +searches environment variables for default values. Set the following +environment variables in your shell: ```shell export NOMAD_ADDR=https://localhost:4646 export NOMAD_CACERT=nomad-ca.pem -export NOMAD_CLIENT_CERT=client.pem -export NOMAD_CLIENT_KEY=client-key.pem +export NOMAD_CLIENT_CERT=cli.pem +export NOMAD_CLIENT_KEY=cli-key.pem ``` -The `NOMAD_CACERT` also needs to be set so the CLI can verify it's talking to -an actual Nomad node. Finally, `NOMAD_CLIENT_CERT` and `NOMAD_CLIENT_KEY` need -to be set since we enabled `verify_https_client` above which prevents any -access lacking a client certificate. +* `NOMAD_ADDR` is the URL of the Nomad agent and sets the default for `-addr`. +* `NOMAD_CACERT` is the location of your CA certificate and sets the default + for `-ca-cert`. +* `NOMAD_CLIENT_CERT` is the location of your CLI certificate and sets the + default for `-client-cert`. +* `NOMAD_CLIENT_KEY` is the location of your CLI key and sets the default for + `-client-key`. -Now the CLI works as expected: +After these environment variables are correctly configured, the CLI will +respond as expected: ```text -vagrant@nomad:~$ nomad node-status +$ nomad node-status ID DC Name Class Drain Status 237cd4c5 dc1 nomad false ready -vagrant@nomad:~$ nomad init +$ nomad init Example job file written to example.nomad vagrant@nomad:~$ nomad run example.nomad ==> Monitoring evaluation "e9970e1d" @@ -312,64 +344,83 @@ vagrant@nomad:~$ nomad run example.nomad ## Server Gossip -We haven't quite completely secured Nomad's communications: Nomad server's -gossip protocol uses a shared key instead of TLS for encryption. This -encryption key must be added to every server's configuration using the -[`encrypt`](/docs/agent/configuration/server.html#encrypt) parameter. +At this point all of Nomad's RPC and HTTP communication is secured with mTLS. +However, Nomad servers also communicate with a gossip protocol, Serf, that does +not use TLS: + +* HTTP - Used to communicate between CLI and Nomad agents. Secured by mTLS. +* RPC - Used to communicate between Nomad agents. Secured by mTLS. +* Serf - Used to communicate between Nomad servers. Secured by a shared key. -As a convenience the Nomad CLI includes a `keygen` command for generating a new -secure gossip encryption key: +Nomad server's gossip protocol use a shared key instead of TLS for encryption. +This encryption key must be added to every server's configuration using the +[`encrypt`](/docs/agent/configuration/server.html#encrypt) parameter or with +the [`-encrypt` command line option](/docs/commands/agent.html). + +The Nomad CLI includes a `keygen` command for generating a new secure gossip +encryption key: ```text $ nomad keygen cg8StVXbQJ0gPvMd9o7yrg== ``` -Put the generated key into each server's configuration file: +Alternatively, you can use any method that base64 encodes 16 random bytes: + +```text +$ dd if=/dev/urandom bs=16 count=1 status=none | base64 +LsuYyj93KVfT3pAJPMMCgA== +$ python -c 'import base64; print base64.b64encode(open("/dev/urandom").read(16))' +uTI2KkW+5WrRTETEfc0ZBQ== +``` + +Put the same generated key into every server's configuration file or command +line arguments: ```hcl server { - enabled = true + enabled = true - # Self-elect, should be 3 or 5 for production - bootstrap_expect = 1 + # Self-elect, should be 3 or 5 for production + bootstrap_expect = 1 - # Encrypt gossip communication - encrypt = "cg8StVXbQJ0gPvMd9o7yrg==" + # Encrypt gossip communication + encrypt = "cg8StVXbQJ0gPvMd9o7yrg==" } ``` ## Switching an existing cluster to TLS -Since Nomad does *not* use different ports for TLS and non-TLS communication, +Since Nomad does _not_ use different ports for TLS and non-TLS communication, the use of TLS should be consistent across the cluster. Switching an existing cluster to use TLS everywhere is similar to upgrading between versions of -Nomad. +Nomad: -First make sure all of your nodes are ready to be switched: - -* Add the appropriate key and certificates to all nodes. +1. Add the appropriate key and certificates to all nodes. * Ensure the private key file is only readable by the Nomad user. -* Add the environment variables to all nodes where the CLI is used. -* Add the appropriate `tls` block to the configuration file on all nodes. -* Generate a gossip key and add it the Nomad server configuration. +1. Add the environment variables to all nodes where the CLI is used. +1. Add the appropriate `tls` block to the configuration file on all nodes. +1. Generate a gossip key and add it the Nomad server configuration. At this point a rolling restart of the cluster will enable TLS everywhere. 1. Restart servers, one at a time -2. Restart clients, one or more at a time +1. Restart clients, one or more at a time -~> As soon as a quorum of servers are TLS-enabled, clients will not be able to - communicate with them until they are restarted. +~> Once a quorum of servers are TLS-enabled, clients will no longer be able to + communicate with the servers until their client configuration is updated and + reloaded. -Jobs running in the cluster will *not* be affected and will continue running +Jobs running in the cluster will _not_ be affected and will continue running throughout the switch. -[guide-server]: https://raw.githubusercontent.com/hashicorp/nomad/master/demo/vagrant/server.hcl -[guide-cluster]: https://www.nomadproject.io/intro/getting-started/cluster.html -[letsencrypt]: https://letsencrypt.org/ [cfssl]: https://cfssl.org/ [cfssl.json]: https://raw.githubusercontent.com/hashicorp/nomad/master/demo/vagrant/cfssl.json -[Vagrantfile]: https://raw.githubusercontent.com/hashicorp/nomad/master/demo/vagrant/Vagrantfile -[Vault]: https://www.vaultproject.io/ +[guide-install]: https://www.nomadproject.io/intro/getting-started/install.html +[guide-cluster]: https://www.nomadproject.io/intro/getting-started/cluster.html +[guide-server]: https://raw.githubusercontent.com/hashicorp/nomad/master/demo/vagrant/server.hcl +[letsencrypt]: https://letsencrypt.org/ +[tls]: https://en.wikipedia.org/wiki/Transport_Layer_Security +[vagrantfile]: https://raw.githubusercontent.com/hashicorp/nomad/master/demo/vagrant/Vagrantfile +[vault]: https://www.vaultproject.io/ [vault-pki]: https://www.vaultproject.io/docs/secrets/pki/index.html From 3ff3d3e95e70a88044bde750e9eabafa5202c36e Mon Sep 17 00:00:00 2001 From: Michael Schurter Date: Tue, 1 Aug 2017 16:50:37 -0700 Subject: [PATCH 07/13] Missed a mention of 127.0.0.1 as a SAN --- website/source/guides/tls.html.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/website/source/guides/tls.html.md b/website/source/guides/tls.html.md index 14ef2fcf039..a307d60572a 100644 --- a/website/source/guides/tls.html.md +++ b/website/source/guides/tls.html.md @@ -118,10 +118,10 @@ echo '{}' | cfssl gencert -ca nomad-ca.pem -ca-key nomad-ca-key.pem -profile=cli - | cfssljson -bare cli ``` -Using `localhost` as a subject alternate name (SAN) allows tools like `curl` to -be able to communicate with Nomad's HTTP API when run on the same host. Other -SANs may be added including a DNS resolvable hostname to allow remote HTTP -requests from third party tools. +Using `localhost` and `127.0.0.1` as subject alternate names (SANs) allows +tools like `curl` to be able to communicate with Nomad's HTTP API when run on +the same host. Other SANs may be added including a DNS resolvable hostname to +allow remote HTTP requests from third party tools. You should now have the following files: From 3bb3ff77ee79654812d655ea5aac802bfb0b90eb Mon Sep 17 00:00:00 2001 From: Michael Schurter Date: Tue, 1 Aug 2017 17:06:56 -0700 Subject: [PATCH 08/13] Sort filenames --- website/source/guides/tls.html.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/source/guides/tls.html.md b/website/source/guides/tls.html.md index a307d60572a..c14b17415d7 100644 --- a/website/source/guides/tls.html.md +++ b/website/source/guides/tls.html.md @@ -130,8 +130,8 @@ You should now have the following files: * `nomad-ca-key.pem` - CA private key. Keep safe! * `nomad-ca.pem` - CA public certificate. * `cli.csr` - Nomad CLI certificate signing request. -* `cli.pem` - Nomad CLI certificate. * `cli-key.pem` - Nomad CLI private key. +* `cli.pem` - Nomad CLI certificate. * `client.csr` - Nomad client node certificate signing request for the `global` region. * `client-key.pem` - Nomad client node private key for the `global` region. * `client.pem` - Nomad client node public certificate for the `global` region. From 295ef10c8d880a369a6dda92cffbad32280dfafc Mon Sep 17 00:00:00 2001 From: Michael Schurter Date: Tue, 1 Aug 2017 17:13:37 -0700 Subject: [PATCH 09/13] Reflow --- website/source/guides/tls.html.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/website/source/guides/tls.html.md b/website/source/guides/tls.html.md index c14b17415d7..0dcfdd2b681 100644 --- a/website/source/guides/tls.html.md +++ b/website/source/guides/tls.html.md @@ -147,8 +147,7 @@ public certificate (`nomad-ca.pem`). Next Nomad must be configured to use the newly-created key and certificates for mTLS. Starting with the [server configuration from the Getting Started -guide][guide-server] add the following TLS -configuration options: +guide][guide-server] add the following TLS configuration options: ```hcl # Increase log verbosity From a7346e186f8de2bc171d6e34ebf7a06267027a5d Mon Sep 17 00:00:00 2001 From: Michael Schurter Date: Tue, 1 Aug 2017 17:37:13 -0700 Subject: [PATCH 10/13] Add $ prefix to shell examples --- website/source/guides/tls.html.md | 32 +++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/website/source/guides/tls.html.md b/website/source/guides/tls.html.md index 0dcfdd2b681..ce3e660993a 100644 --- a/website/source/guides/tls.html.md +++ b/website/source/guides/tls.html.md @@ -60,8 +60,8 @@ use [cfssl][cfssl]. You can generate a private CA certificate and key with [cfssl][cfssl]: ```shell -# Generate the CA's private key and certificate -cfssl print-defaults csr | cfssl gencert -initca - | cfssljson -bare nomad-ca +$ # Generate the CA's private key and certificate +$ cfssl print-defaults csr | cfssl gencert -initca - | cfssljson -bare nomad-ca ``` The CA key (`nomad-ca-key.pem`) will be used to sign certificates for Nomad @@ -105,16 +105,16 @@ increase the default certificate expiration time: ``` ```shell -# Generate a certificate for the Nomad server -echo '{}' | cfssl gencert -ca=nomad-ca.pem -ca-key=nomad-ca-key.pem -config=cfssl.json \ +$ # Generate a certificate for the Nomad server +$ echo '{}' | cfssl gencert -ca=nomad-ca.pem -ca-key=nomad-ca-key.pem -config=cfssl.json \ -hostname="server.global.nomad,localhost,127.0.0.1" - | cfssljson -bare server # Generate a certificate for the Nomad client -echo '{}' | cfssl gencert -ca=nomad-ca.pem -ca-key=nomad-ca-key.pem -config=cfssl.json \ +$ echo '{}' | cfssl gencert -ca=nomad-ca.pem -ca-key=nomad-ca-key.pem -config=cfssl.json \ -hostname="client.global.nomad,localhost,127.0.0.1" - | cfssljson -bare client # Generate a certificate for the CLI -echo '{}' | cfssl gencert -ca nomad-ca.pem -ca-key nomad-ca-key.pem -profile=client \ +$ echo '{}' | cfssl gencert -ca nomad-ca.pem -ca-key nomad-ca-key.pem -profile=client \ - | cfssljson -bare cli ``` @@ -282,16 +282,16 @@ server we can test our TLS-enabled cluster! In separate terminals start a server and client agent: ```shell -# In one terminal... -nomad agent -config server1.hcl +$ # In one terminal... +$ nomad agent -config server1.hcl -# ...and in another -nomad agent -config client1.hcl +$ # ...and in another +$ nomad agent -config client1.hcl ``` If you run `nomad node-status` now, you'll get an error, like: -``` +```text Error querying node status: Get http://127.0.0.1:4646/v1/nodes: malformed HTTP response "\x15\x03\x01\x00\x02\x02" ``` @@ -300,7 +300,7 @@ HTTPS. We can configure the local Nomad client to connect using TLS and specify our custom keys and certificates using the command line: ```shell -nomad node-status -ca-cert=nomad-ca.pem -client-cert=cli.pem -client-key=cli-key.pem -addr=https://127.0.0.1:4646 +$ nomad node-status -ca-cert=nomad-ca.pem -client-cert=cli.pem -client-key=cli-key.pem -addr=https://127.0.0.1:4646 ``` This process can be cumbersome to type each time, so the Nomad CLI also @@ -308,10 +308,10 @@ searches environment variables for default values. Set the following environment variables in your shell: ```shell -export NOMAD_ADDR=https://localhost:4646 -export NOMAD_CACERT=nomad-ca.pem -export NOMAD_CLIENT_CERT=cli.pem -export NOMAD_CLIENT_KEY=cli-key.pem +$ export NOMAD_ADDR=https://localhost:4646 +$ export NOMAD_CACERT=nomad-ca.pem +$ export NOMAD_CLIENT_CERT=cli.pem +$ export NOMAD_CLIENT_KEY=cli-key.pem ``` * `NOMAD_ADDR` is the URL of the Nomad agent and sets the default for `-addr`. From 69d561e15e7bcc71f395b2e023e365467b0c9cd1 Mon Sep 17 00:00:00 2001 From: Michael Schurter Date: Fri, 11 Aug 2017 20:20:11 -0700 Subject: [PATCH 11/13] Add openssl rand example; reword intermediate CAs Thanks @jvoorhis! --- website/source/guides/tls.html.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/website/source/guides/tls.html.md b/website/source/guides/tls.html.md index ce3e660993a..bef4b49844b 100644 --- a/website/source/guides/tls.html.md +++ b/website/source/guides/tls.html.md @@ -48,9 +48,8 @@ be signed by the same Certificate Authority (CA). This should be a _private_ CA and not a public one like [Let's Encrypt][letsencrypt] as any certificate signed by this CA will be allowed to communicate with the cluster. -~> Nomad certificates may be signed by different intermediate CAs as long as - the full `ca_file` on each node contains all of the CA certificates in the - chain. +~> Nomad certificates may be signed by intermediate CAs as long as the root CA + is the same. Append all intermediate CAs to the `cert_file`. ### Certificate Authority @@ -367,10 +366,10 @@ cg8StVXbQJ0gPvMd9o7yrg== Alternatively, you can use any method that base64 encodes 16 random bytes: ```text +$ openssl rand -base64 16 +raZjciP8vikXng2S5X0m9w== $ dd if=/dev/urandom bs=16 count=1 status=none | base64 LsuYyj93KVfT3pAJPMMCgA== -$ python -c 'import base64; print base64.b64encode(open("/dev/urandom").read(16))' -uTI2KkW+5WrRTETEfc0ZBQ== ``` Put the same generated key into every server's configuration file or command From bec5519126ae092398ae4929029b55b03cc8f311 Mon Sep 17 00:00:00 2001 From: Michael Schurter Date: Fri, 11 Aug 2017 20:57:59 -0700 Subject: [PATCH 12/13] Oops! Forgot demo/vagrant/cfssl.json file --- demo/vagrant/cfssl.json | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 demo/vagrant/cfssl.json diff --git a/demo/vagrant/cfssl.json b/demo/vagrant/cfssl.json new file mode 100644 index 00000000000..6e438c9b9d6 --- /dev/null +++ b/demo/vagrant/cfssl.json @@ -0,0 +1,13 @@ +{ + "signing": { + "default": { + "expiry": "87600h", + "usages": [ + "signing", + "key encipherment", + "server auth", + "client auth" + ] + } + } +} From cadbe7a77f3ef46f6afd78e695f9ecec6ecbb0e3 Mon Sep 17 00:00:00 2001 From: Michael Schurter Date: Fri, 11 Aug 2017 21:00:13 -0700 Subject: [PATCH 13/13] Rename guide from tls to securing-nomad --- website/source/guides/{tls.html.md => securing-nomad.html.md} | 2 +- website/source/layouts/guides.erb | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) rename website/source/guides/{tls.html.md => securing-nomad.html.md} (99%) diff --git a/website/source/guides/tls.html.md b/website/source/guides/securing-nomad.html.md similarity index 99% rename from website/source/guides/tls.html.md rename to website/source/guides/securing-nomad.html.md index bef4b49844b..e0825a6ea24 100644 --- a/website/source/guides/tls.html.md +++ b/website/source/guides/securing-nomad.html.md @@ -1,7 +1,7 @@ --- layout: "guides" page_title: "Securing Nomad with TLS" -sidebar_current: "guides-tls" +sidebar_current: "guides-securing-nomad" description: |- Securing Nomad's cluster communication with TLS is important for both security and easing operations. Nomad can use mutual TLS (mTLS) for diff --git a/website/source/layouts/guides.erb b/website/source/layouts/guides.erb index c844c058349..d53131bec2d 100644 --- a/website/source/layouts/guides.erb +++ b/website/source/layouts/guides.erb @@ -50,8 +50,8 @@ - > - Configuring TLS + > + Securing Nomad >