Cluster failed to start with 2.18 RC5 docker image

[ruanyl]

Using self-signed certificates, working fine for 2.17, but when creating cluster with 2.18 RC5 docker image, the cluster failed to start with the following error, am I missing anything?

java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
Likely root cause: java.security.cert.CertificateExpiredException: NotAfter: Thu Sep 28 04:20:03 UTC 2023
	at java.base/sun.security.x509.CertificateValidity.valid(CertificateValidity.java:182)
	at java.base/sun.security.x509.X509CertImpl.checkValidity(X509CertImpl.java:534)
	at java.base/sun.security.x509.X509CertImpl.checkValidity(X509CertImpl.java:507)
	at org.opensearch.security.ssl.config.KeyStoreUtils.validateKeyStoreCertificates(KeyStoreUtils.java:147)
	at org.opensearch.security.ssl.config.TrustStoreConfiguration.createTrustManagerFactory(TrustStoreConfiguration.java:61)
	at org.opensearch.security.ssl.SslConfiguration.lambda$buildServerSslContext$0(SslConfiguration.java:84)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:571)
	at org.opensearch.security.ssl.SslConfiguration.buildServerSslContext(SslConfiguration.java:73)
	at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:42)
	at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:38)
	at org.opensearch.security.ssl.SslSettingsManager.lambda$buildSslContexts$0(SslSettingsManager.java:96)
	at java.base/java.util.Optional.ifPresentOrElse(Optional.java:196)
	at org.opensearch.security.ssl.SslSettingsManager.buildSslContexts(SslSettingsManager.java:95)
	at org.opensearch.security.ssl.SslSettingsManager.<init>(SslSettingsManager.java:80)
	at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:249)
	at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:318)
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796)
	at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744)
	at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545)
	at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197)
	at org.opensearch.node.Node.<init>(Node.java:523)
	at org.opensearch.node.Node.<init>(Node.java:450)
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181)
	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172)

[DarshitChanpura]

@ruanyl I wasn't able to reproduce with RC5 docker image listed here.

Expand to see docker-compose.yml I used

version: '3'
services:
  opensearch-node1:
    image: opensearchstaging/opensearch:2.18.0.10466
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node1
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
      # - "plugins.security.disabled=true"
      - "OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - opensearch-net
  opensearch-node2:
    image: opensearchstaging/opensearch:2.18.0.10466
    container_name: opensearch-node2
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node2
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
      # - "plugins.security.disabled=true"
      - "OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    networks:
      - opensearch-net
  opensearch-dashboards:
    image: opensearchstaging/opensearch-dashboards:2.18.0.8047
    container_name: opensearch-dashboards
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]'
    networks:
      - opensearch-net

networks:
  opensearch-net:

[DarshitChanpura]

This indicates that your self-signed certificates have expired could you please generate new ones. Closing this as it is not an issue.

[elmbrain]

The expired certificate was in the file in the settings

plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem

You can quickly check everything through

while openssl x509 -noout -text; do:; done < root-ca.pem | grep After

Then find which ones have expired and delete PEM section from the file

remove block of expired certificate from root-ca.pem
Block like this

---BEGIN CERTIFICATE----
adkdlkag
---END CERTIFICATE----

find the block you need and remove it from the file

The real problem is that the log does not indicate which certificate has expired; you have to look for it manually. The problem started with the fact that the validator added a check for trusted certificates, which is generally unnecessary