[BUG] Likely root cause: java.lang.RuntimeException: Demo certificates found

[danpawlik]

Describe the bug
On starting container, opensearch if raising an error:

Aug 18 07:13:52 elasticsearch.sftests.com opensearch[468870]: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
Aug 18 07:13:52 elasticsearch.sftests.com opensearch[468870]: Likely root cause: java.lang.RuntimeException: Demo certificates found [d14aefe70a592d7a29e14f3ff89c3d0070c99e87d21776aa07d333ee877e758f, 54a70016e0837a2b0c5658d1032d7ca32e432c62c55f01a2bf5adcb69a0a7ba9, bdc141ab2272c779d0f242b79063152c49e1b06a2af05e0fd90d505f2b44d5f5, 3e839e2b059036a99ee4f742814995f2fb0ced7e9d68a47851f43a3c630b5324, 9b13661c073d864c28ad7b13eda67dcb6cbc2f04d116adc7c817c20b4c7ed361]

then it quit.

To Reproduce
Steps to reproduce the behavior:

1. Start container with example parameters:

  podman create --name opensearch --net host --pid host \
    -v DISABLE_INSTALL_DEMO_CONFIG=true \
    -v cluster.name=elasticsearch_sftests_coma \
    -v node.name=elk \
    -v discovery.seed_hosts=elk \
    -v cluster.initial_master_nodes=elk \
    -v "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" \
    -v /mnt/opensearch/data:/usr/share/opensearch/data:Z \
    -v /mnt/opensearch/logs:/var/log/opensearch:Z \
    -v /etc/opensearch/certs/localCA.pem:/usr/share/opensearch/config/localCA.pem:Z \
    -v /etc/opensearch/certs/elasticsearch-admin.crt:/usr/share/opensearch/config/elasticsearch-admin.crt:Z \
    -v /etc/opensearch/certs/elasticsearch-admin.key:/usr/share/opensearch/config/elasticsearch-admin.key:Z \
    -v /etc/opensearch/opensearch.yml:/usr/share/opensearch/config/opensearch.yml:Z \
    -v /etc/opensearch/internal_users.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml:Z \
    -v /etc/opensearch/roles_mapping.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/roles_mapping.yml:Z \
    -v /etc/opensearch/tenants.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/tenants.yml:Z \
    -v /etc/opensearch/roles.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/roles.yml:Z \
    -v /etc/opensearch/action_groups.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/action_groups.yml:Z
    opensearchproject/opensearch:1.0.0

opensearch.yml

cluster.name: elasticsearch_sftests_com
node.name: elk
path.data: /usr/share/opensearch/data
path.logs: /var/log/opensearch
network.host: XXX.XXX.XXX.XXX
http.port: 9200

node.master: true

cluster.routing.allocation.disk.threshold_enabled: false
node.max_local_storage_nodes: 3

plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.allow_default_init_securityindex: true
# No matter if plugins.security.allow_unsafe_democertificates is set to true or false,
# problem will raise.
plugins.security.allow_unsafe_democertificates: false

plugins.security.ssl.transport.pemcert_filepath: /usr/share/opensearch/config/elasticsearch-admin.crt
plugins.security.ssl.transport.pemkey_filepath: /usr/share/opensearch/config/elasticsearch-admin.key
plugins.security.ssl.transport.pemtrustedcas_filepath: /usr/share/opensearch/config/localCA.pem
plugins.security.ssl.http.pemcert_filepath: /usr/share/opensearch/config/elasticsearch-admin.crt
plugins.security.ssl.http.pemkey_filepath: /usr/share/opensearch/config/elasticsearch-admin.key
plugins.security.ssl.http.pemtrustedcas_filepath: /usr/share/opensearch/config/localCA.pem
plugins.security.authcz.admin_dn:
  - CN=elasticsearch.sftests.com,O=ElasticsearchSoftwareFactory,L=WROCLAW,ST=WROCLAW,C=PL

plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true

Expected behavior
Service will start normally

Host/Environment (please complete the following information):

• OS: Linux Centos Stream
• Version 1.0.0

Additional context
Environment variable DISABLE_INSTALL_DEMO_CONFIG seems not to work. The demo certificates are still in the
container image.

[nebulon42]

I'm also seeing this and I set the suggested env var DISABLE_INSTALL_DEMO_CONFIG as per documentation in https://opensearch.org/docs/opensearch/install/docker-security but then found out that this does not change anything.

You also cannot use the mentioned workaround in opendistro-for-elasticsearch/opendistro-build#10 as the demo files are seemingly baked into the image.

[nebulon42]

I was only able to get past this by creating a new container image where the demo files have been removed. Mounting a new entrypoint where I remove the files did not work for me. Quite tedious. I think this requires hotfixing as production use of the current images is not possible.

[peterzhuamazon]

Seems like related to #240 as well?

[peterzhuamazon]

Also #369

[peterzhuamazon]

Will revert this so user can skip demo config during start of the container:
https://github.com/opendistro-for-elasticsearch/opendistro-build/blob/aabb00fb349e1eb1b916ed68c22fb5ef8be8abc7/elasticsearch/docker/build/elasticsearch/bin/docker-entrypoint.sh#L85

[peterzhuamazon]

Hi @danpawlik, Let me know if this is fixed with #436.
Please close the issue if it fixed.
Thanks

[danpawlik]

Thanks @peterzhuamazon , it works.

[onelapahead]

This did not get release as part of #151

I'm still seeing the demo certs in the 1.0.1 image, can new images be released ASAP?

[peterzhuamazon]

This did not get release as part of #151

I'm still seeing the demo certs in the 1.0.1 image, can new images be released ASAP?

1.0.1 release on 09/01 and the fix is after that.
We will release this on 1.1.0 version.