[BUG] Docker image not respecting environment variables

[aetter]

Describe the bug

With Docker, you have the option of overriding default values in opensearch_dashboards.yml with environment variables. That way, instead of passing a new opensearch_dashboards.yml file to the container, you can just specify a few lines in your Docker Compose file. OpenSearch Dashboards appears to be ignoring environment variables currently.

To Reproduce

1. Copy the following Docker Compose file to your machine as docker-compose.yml:

version: '3'
services:
  opensearch-node1:
    image: opensearchstaging/opensearch:latest
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node1
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_master_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the Elasticsearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - opensearch-net
  opensearch-node2:
    image: opensearchstaging/opensearch:latest
    container_name: opensearch-node2
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node2
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_master_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch-data2:/usr/share/opensearch/data
    networks:
      - opensearch-net
  opensearch-dashboards:
    image: opensearchstaging/opensearch-dashboards:latest
    container_name: opensearch-dashboards
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      OPENSEARCH_HOSTS: https://opensearch-node1:9200
    networks:
      - opensearch-net

volumes:
  opensearch-data1:
  opensearch-data2:

networks:
  opensearch-net:

2. From its directory, run docker-compose up.
3. After the cluster stabilizes, note that OpenSearch Dashboards won't connect to it. Instead, it repeatedly tries to connect to https://localhost:9200, as specified in the container's opensearch_dashboards.yml:

opensearch-dashboards    | {"type":"log","@timestamp":"2021-05-04T22:42:13Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}

4. Run docker-compose down -v to stop the cluster.
5. Replace your docker-compose.yml with this:

version: '3'
services:
  opensearch-node1:
    image: opensearchstaging/opensearch:latest
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node1
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_master_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the Elasticsearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - opensearch-net
  opensearch-node2:
    image: opensearchstaging/opensearch:latest
    container_name: opensearch-node2
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node2
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_master_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch-data2:/usr/share/opensearch/data
    networks:
      - opensearch-net
  opensearch-dashboards:
    image: opensearchstaging/opensearch-dashboards:latest
    container_name: opensearch-dashboards
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      OPENSEARCH_HOSTS: https://opensearch-node1:9200
    volumes:
      - ./sample-opensearch_dashboards.yml:/usr/share/opensearch-dashboards/config/opensearch_dashboards.yml
    networks:
      - opensearch-net

volumes:
  opensearch-data1:
  opensearch-data2:

networks:
  opensearch-net:

Note that there's now a volumes section to copy a new file over to the container.

6. Create sample-opensearch_dashboards.yml:

server.name: "opensearch-dashboards"
server.host: "0"
opensearch.hosts: ["https://opensearch-node1:9200"]
opensearch.ssl.verificationMode: none
opensearch.username: "kibanaserver"
opensearch.password: "kibanaserver"
opensearch.requestHeadersWhitelist: ["securitytenant", "Authorization"]

opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: ["Private", "Global"]
opensearch_security.readonly_mode.roles: ["kibana_read_only"]
# Use this setting if you are running kibana without https
opensearch_security.cookie.secure: false

7. Run docker-compose up again.
8. Note that it works this time.

Expected behavior

Environment variables from Docker should override any settings in opensearch_dashboards.yml, provided they are specified in ALL_CAPS with underscores to separate words.

OpenSearch Version
1.0.0-beta1

Dashboards Version
1.0.0-beta1

Plugins

opensearch-node1 opensearch-alerting             1.0.0.0-beta1
opensearch-node1 opensearch-anomaly-detection    1.0.0.0-beta1
opensearch-node1 opensearch-asynchronous-search  1.0.0.0-beta1
opensearch-node1 opensearch-index-management     1.0.0.0-beta1
opensearch-node1 opensearch-job-scheduler        1.0.0.0-beta1
opensearch-node1 opensearch-knn                  1.0.0.0-beta1
opensearch-node1 opensearch-performance-analyzer 1.0.0.0-beta1
opensearch-node1 opensearch-reports-scheduler    1.0.0.0-beta1
opensearch-node1 opensearch-security             1.0.0.0-beta1
opensearch-node1 opensearch-sql                  1.0.0.0-beta1

Host/Environment (please complete the following information):

• OS: macOS
• Browser and version: N/A

[dblock]

@aetter What was the resolution here?

[aetter]

@dblock I pulled the image around noon, and independent of this bug report, @peterzhuamazon pushed a new image with a fix a few hours later.