Support PKCS#11 tokens as keystores and truststores

docs/reference/settings/security-settings.asciidoc

@@ -400,8 +400,9 @@ The path to the Java Keystore file that contains a private key and certificate.
 `ssl.key` and `ssl.keystore.path` may not be used at the same time.
 
 `ssl.keystore.type`::
-The format of the keystore file. Should be either `jks` to use the Java
-Keystore format, or `PKCS12` to use PKCS#12 files. The default is `jks`.
+The format of the keystore file. Should be `jks` to use the Java
+Keystore format, `PKCS12` to use PKCS#12 files, or `PKCS11` to use a PKCS#11 token.
+The default is `jks`.
 
 `ssl.keystore.password`::
 The password to the keystore.
@@ -426,8 +427,9 @@ The password to the truststore.
 The password to the truststore.
 
 `ssl.truststore.type`::
-The format of the keystore file. Should be either `jks` to use the Java
-Keystore format, or `PKCS12` to use PKCS#12 files. The default is `jks`.
+The format of the keystore file. Should be `jks` to use the Java
+Keystore format, `PKCS12` to use PKCS#12 files, or `PKCS11` to use a PKCS#11 token.
+The default is `jks`.
 
 `ssl.verification_mode`::
 Indicates the type of verification when using `ldaps` to protect against man
@@ -649,8 +651,9 @@ The path to the Java Keystore file that contains a private key and certificate.
 `ssl.key` and `ssl.keystore.path` cannot be used at the same time.
 
 `ssl.keystore.type`::
-The format of the keystore file. Should be either `jks` to use the Java
-Keystore format, or `PKCS12` to use PKCS#12 files. The default is `jks`.
+The format of the keystore file. Should be `jks` to use the Java
+Keystore format, `PKCS12` to use PKCS#12 files, or `PKCS11` to use a PKCS#11 token.
+The default is `jks`.
 
 `ssl.truststore.password`::
 The password to the truststore.
@@ -664,8 +667,9 @@ The path to the Java Keystore file that contains the certificates to trust.
 same time.
 
 `ssl.truststore.type`::
-The format of the truststore file. Should be either `jks` to use the Java
-Keystore format, or `PKCS12` to use PKCS#12 files. The default is `jks`.
+The format of the truststore file. Should be `jks` to use the Java
+Keystore format, `PKCS12` to use PKCS#12 files, or `PKCS11` to use a PKCS#11 token.
+The default is `jks`.
 
 `ssl.verification_mode`::
 Indicates the type of verification when using `ldaps` to protect against man
@@ -1062,7 +1066,7 @@ Must be either a Java Keystore (jks) or a PKCS#12 file.
 same time.
 
 `ssl.truststore.type`::
-The type of the truststore (`ssl.truststore.path`). Must be either `jks` or 
+The type of the truststore (`ssl.truststore.path`). Must be either `jks` or
 `PKCS12`. If the keystore path ends in ".p12", ".pfx" or "pkcs12", this setting 
 defaults to `PKCS12`. Otherwise, it defaults to `jks`.
 
@@ -1316,6 +1320,29 @@ a PKCS#12 container includes trusted certificate ("anchor") entries look for
 `openssl pkcs12 -info` output, or `trustedCertEntry` in the
 `keytool -list` output.
 
+===== PKCS#11 tokens
+
+When using a PKCS#11 cryptographic token, which contain the
+private key, certificate, and certificates that should be trusted, use
+the following settings:
+
+`xpack.ssl.keystore.type`::
+Set this to `PKCS11`.
+
+`xpack.ssl.truststore.type`::
+Set this to `PKCS11`.
+
+
+[[pkcs11-truststore-note]]
+[NOTE]
+When configuring the PKCS#11 token that your JVM is configured to use as
+a keystore or a truststore for Elasticsearch, the PIN for the token needs
+to be configured via the appropriate JVM setting (`-Djavax.net.ssl.trustStorePassword`).
+Since there can only be one PKCS#11 token configured, only one keystore and
+truststore will be usable for configuration in Elasticsearch. This in turn means
+that only one certificate can be used for TLS both in the transport and the
+http layer.
+
 [[http-tls-ssl-settings]]
 :ssl-prefix:             xpack.security.http
 :component:              HTTP

docs/reference/settings/ssl-settings.asciidoc

@@ -145,3 +145,17 @@ Password to the PKCS#12 file.
 
 +{ssl-prefix}.ssl.truststore.secure_password+ (<<secure-settings,Secure>>)::
 Password to the PKCS#12 file.
+
+===== PKCS#11 Tokens
+
+{security} can be configured to use a PKCS#11 token that contains the private key,
+certificate and certificates that should be trusted.
+
+PKCS#11 token require additional configuration on the JVM level and can be enabled
+via the following settings:
+
++{ssl-prefix}.keystore.type+::
+Set this to `PKCS11` to indicate that the PKCS#11 token should be used as a keystore.
+
++{ssl-prefix}.truststore.type+::
+Set this to `PKCS11` to indicate that the PKCS#11 token should be used as a truststore.

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ssl/CertParsingUtils.java

@@ -197,6 +197,7 @@ public static X509ExtendedKeyManager getKeyManager(X509KeyPairSettings keyPair,
     static KeyConfig createKeyConfig(X509KeyPairSettings keyPair, Settings settings, String trustStoreAlgorithm) {
         String keyPath = keyPair.keyPath.get(settings).orElse(null);
         String keyStorePath = keyPair.keystorePath.get(settings).orElse(null);
+        String keyStoreType = getKeyStoreType(keyPair.keystoreType, settings, keyStorePath);
 
         if (keyPath != null && keyStorePath != null) {
             throw new IllegalArgumentException("you cannot specify a keystore and key file");
@@ -212,10 +213,9 @@ static KeyConfig createKeyConfig(X509KeyPairSettings keyPair, Settings settings,
             return new PEMKeyConfig(keyPath, keyPassword, certPath);
         }
 
-        if (keyStorePath != null) {
+        if (keyStorePath != null || keyStoreType.equalsIgnoreCase("pkcs11")) {
             SecureString keyStorePassword = keyPair.keystorePassword.get(settings);
             String keyStoreAlgorithm = keyPair.keystoreAlgorithm.get(settings);
-            String keyStoreType = getKeyStoreType(keyPair.keystoreType, settings, keyStorePath);
             SecureString keyStoreKeyPassword = keyPair.keystoreKeyPassword.get(settings);
             if (keyStoreKeyPassword.length() == 0) {
                 keyStoreKeyPassword = keyStorePassword;
@@ -224,7 +224,6 @@ static KeyConfig createKeyConfig(X509KeyPairSettings keyPair, Settings settings,
                     trustStoreAlgorithm);
         }
         return null;
-
     }
 
     /**

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ssl/SSLConfiguration.java

@@ -212,12 +212,11 @@ private static TrustConfig createTrustConfig(Settings settings, KeyConfig keyCon
 
     private static TrustConfig createCertChainTrustConfig(Settings settings, KeyConfig keyConfig, SSLConfiguration global) {
         String trustStorePath = SETTINGS_PARSER.truststorePath.get(settings).orElse(null);
-
+        String trustStoreType = getKeyStoreType(SETTINGS_PARSER.truststoreType, settings, trustStorePath);
         List<String> caPaths = getListOrNull(SETTINGS_PARSER.caPaths, settings);
         if (trustStorePath != null && caPaths != null) {
             throw new IllegalArgumentException("you cannot specify a truststore and ca files");
         }
-
         VerificationMode verificationMode = SETTINGS_PARSER.verificationMode.get(settings).orElseGet(() -> {
             if (global != null) {
                 return global.verificationMode();
@@ -228,16 +227,15 @@ private static TrustConfig createCertChainTrustConfig(Settings settings, KeyConf
             return TrustAllConfig.INSTANCE;
         } else if (caPaths != null) {
             return new PEMTrustConfig(caPaths);
-        } else if (trustStorePath != null) {
+        } else if (trustStorePath != null || trustStoreType.equalsIgnoreCase("pkcs11")) {
             SecureString trustStorePassword = SETTINGS_PARSER.truststorePassword.get(settings);
             String trustStoreAlgorithm = SETTINGS_PARSER.truststoreAlgorithm.get(settings);
-            String trustStoreType = getKeyStoreType(SETTINGS_PARSER.truststoreType, settings, trustStorePath);
             return new StoreTrustConfig(trustStorePath, trustStoreType, trustStorePassword, trustStoreAlgorithm);
         } else if (global == null && System.getProperty("javax.net.ssl.trustStore") != null
             && System.getProperty("javax.net.ssl.trustStore").equals("NONE") == false) {
             try (SecureString truststorePassword = new SecureString(System.getProperty("javax.net.ssl.trustStorePassword", ""))) {
                 return new StoreTrustConfig(System.getProperty("javax.net.ssl.trustStore"), KeyStore.getDefaultType(), truststorePassword,
-                        System.getProperty("ssl.TrustManagerFactory.algorithm", TrustManagerFactory.getDefaultAlgorithm()));
+                    System.getProperty("ssl.TrustManagerFactory.algorithm", TrustManagerFactory.getDefaultAlgorithm()));
             }
         } else if (global != null && keyConfig == global.keyConfig()) {
             return global.trustConfig();

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ssl/StoreKeyConfig.java

@@ -15,8 +15,6 @@
 import javax.net.ssl.X509ExtendedTrustManager;
 
 import java.io.IOException;
-import java.io.InputStream;
-import java.nio.file.Files;
 import java.nio.file.Path;
 import java.security.GeneralSecurityException;
 import java.security.Key;
@@ -49,7 +47,7 @@ class StoreKeyConfig extends KeyConfig {
 
     /**
      * Creates a new configuration that can be used to load key and trust material from a {@link KeyStore}
-     * @param keyStorePath the path to the keystore file
+     * @param keyStorePath the path to the keystore file or null when keyStoreType is pkcs11
      * @param keyStoreType the type of the keystore file
      * @param keyStorePassword the password for the keystore
      * @param keyPassword the password for the private key in the keystore
@@ -58,7 +56,7 @@ class StoreKeyConfig extends KeyConfig {
      */
     StoreKeyConfig(String keyStorePath, String keyStoreType, SecureString keyStorePassword, SecureString keyPassword,
                    String keyStoreAlgorithm, String trustStoreAlgorithm) {
-        this.keyStorePath = Objects.requireNonNull(keyStorePath, "keystore path must be specified");
+        this.keyStorePath = keyStorePath;
         this.keyStoreType = Objects.requireNonNull(keyStoreType, "keystore type must be specified");
         // since we support reloading the keystore, we must store the passphrase in memory for the life of the node, so we
         // clone the password and never close it during our uses below
@@ -71,7 +69,7 @@ class StoreKeyConfig extends KeyConfig {
     @Override
     X509ExtendedKeyManager createKeyManager(@Nullable Environment environment) {
         try {
-            KeyStore ks = getKeyStore(environment);
+            KeyStore ks = getStore(environment, keyStorePath, keyStoreType, keyStorePassword);
             checkKeyStore(ks);
             return CertParsingUtils.keyManager(ks, keyPassword.getChars(), keyStoreAlgorithm);
         } catch (IOException | CertificateException | NoSuchAlgorithmException | UnrecoverableKeyException | KeyStoreException e) {
@@ -82,8 +80,9 @@ X509ExtendedKeyManager createKeyManager(@Nullable Environment environment) {
     @Override
     X509ExtendedTrustManager createTrustManager(@Nullable Environment environment) {
         try {
-            return CertParsingUtils.trustManager(keyStorePath, keyStoreType, keyStorePassword.getChars(), trustStoreAlgorithm, environment);
-        } catch (Exception e) {
+            KeyStore ks = getStore(environment, keyStorePath, keyStoreType, keyStorePassword);
+            return CertParsingUtils.trustManager(ks, trustStoreAlgorithm);
+        } catch (IOException | CertificateException | NoSuchAlgorithmException | KeyStoreException e) {
             throw new ElasticsearchException("failed to initialize a TrustManagerFactory", e);
         }
     }
@@ -112,13 +111,16 @@ Collection<CertificateInfo> certificates(Environment environment) throws General
 
     @Override
     List<Path> filesToMonitor(@Nullable Environment environment) {
+        if (keyStorePath == null) {
+            return Collections.emptyList();
+        }
         return Collections.singletonList(CertParsingUtils.resolvePath(keyStorePath, environment));
     }
 
     @Override
     List<PrivateKey> privateKeys(@Nullable Environment environment) {
         try {
-            KeyStore keyStore = getKeyStore(environment);
+            KeyStore keyStore = getStore(environment, keyStorePath, keyStoreType, keyStorePassword);
             List<PrivateKey> privateKeys = new ArrayList<>();
             for (Enumeration<String> e = keyStore.aliases(); e.hasMoreElements(); ) {
                 final String alias = e.nextElement();
@@ -135,15 +137,6 @@ List<PrivateKey> privateKeys(@Nullable Environment environment) {
         }
     }
 
-    private KeyStore getKeyStore(@Nullable Environment environment)
-                                throws KeyStoreException, CertificateException, NoSuchAlgorithmException, IOException {
-        try (InputStream in = Files.newInputStream(CertParsingUtils.resolvePath(keyStorePath, environment))) {
-            KeyStore ks = KeyStore.getInstance(keyStoreType);
-            ks.load(in, keyStorePassword.getChars());
-            return ks;
-        }
-    }
-
     private void checkKeyStore(KeyStore keyStore) throws KeyStoreException {
         Enumeration<String> aliases = keyStore.aliases();
         while (aliases.hasMoreElements()) {

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ssl/StoreTrustConfig.java

@@ -55,8 +55,8 @@ class StoreTrustConfig extends TrustConfig {
     @Override
     X509ExtendedTrustManager createTrustManager(@Nullable Environment environment) {
         try {
-            return CertParsingUtils.trustManager(trustStorePath, trustStoreType, trustStorePassword.getChars(),
-                    trustStoreAlgorithm, environment);
+            KeyStore trustStore = getStore(environment, trustStorePath, trustStoreType, trustStorePassword);
+            return CertParsingUtils.trustManager(trustStore, trustStoreAlgorithm);
         } catch (Exception e) {
             throw new ElasticsearchException("failed to initialize a TrustManagerFactory", e);
         }

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ssl/TrustConfig.java

@@ -7,14 +7,21 @@
 
 import org.elasticsearch.ElasticsearchException;
 import org.elasticsearch.common.Nullable;
+import org.elasticsearch.common.settings.SecureString;
 import org.elasticsearch.env.Environment;
 import org.elasticsearch.xpack.core.ssl.cert.CertificateInfo;
 
 import javax.net.ssl.X509ExtendedTrustManager;
 
 import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
 import java.nio.file.Path;
 import java.security.GeneralSecurityException;
+import java.security.KeyStore;
+import java.security.KeyStoreException;
+import java.security.NoSuchAlgorithmException;
+import java.security.cert.CertificateException;
 import java.security.cert.X509Certificate;
 import java.util.ArrayList;
 import java.util.Arrays;
@@ -58,6 +65,38 @@ abstract class TrustConfig {
      */
     public abstract int hashCode();
 
+    /**
+     * Loads and returns the appropriate {@link KeyStore} for the given configuration. The KeyStore can be backed by a file
+     * in any format that the Security Provider might support, or a cryptographic software or hardware token in the case
+     * of a PKCS#11 Provider.
+     *
+     * @param environment   the environment to resolve files against or null in the case of running in a transport client
+     * @param storePath     the path to the {@link KeyStore} to load, or null if a PKCS11 token is configured as the keystore/truststore
+     *                      of the JVM
+     * @param storeType     the type of the {@link KeyStore}
+     * @param storePassword the password to be used for decrypting the {@link KeyStore}
+     * @return the loaded KeyStore to be used as a keystore or a truststore
+     * @throws KeyStoreException        if an instance of the specified type cannot be loaded
+     * @throws CertificateException     if any of the certificates in the keystore could not be loaded
+     * @throws NoSuchAlgorithmException if the algorithm used to check the integrity of the keystore cannot be found
+     * @throws IOException              if there is an I/O issue with the KeyStore data or the password is incorrect
+     */
+    KeyStore getStore(@Nullable Environment environment, @Nullable String storePath, String storeType, SecureString storePassword)
+        throws KeyStoreException, CertificateException, NoSuchAlgorithmException, IOException {
+        if (null != storePath) {
+            try (InputStream in = Files.newInputStream(CertParsingUtils.resolvePath(storePath, environment))) {
+                KeyStore ks = KeyStore.getInstance(storeType);
+                ks.load(in, storePassword.getChars());
+                return ks;
+            }
+        } else if (storeType.equalsIgnoreCase("pkcs11")) {
+            KeyStore ks = KeyStore.getInstance(storeType);
+            ks.load(null, storePassword.getChars());
+            return ks;
+        }
+        throw new IllegalArgumentException("keystore.path or truststore.path can only be empty when using a PKCS#11 token");
+    }
+
     /**
      * A trust configuration that is a combination of a trust configuration with the default JDK trust configuration. This trust
      * configuration returns a trust manager verifies certificates against both the default JDK trusted configurations and the specific