[BUG] java.lang.IllegalArgumentException: unknown setting [node] please check that any required plugins are installed

[samirmajen]

Describe the bug
Running the latest version of opensearch (1.1.0) as a helm template in kubernetes gives the following error in the logs:

uncaught exception in thread [main]
java.lang.IllegalArgumentException: unknown setting [node] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:557)
at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:502)
at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:473)
at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:444)
at org.opensearch.common.settings.SettingsModule.(SettingsModule.java:162)
at org.opensearch.node.Node.(Node.java:417)
at org.opensearch.node.Node.(Node.java:301)
at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:240)
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:240)
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:406)
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:182)
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:173)
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:99)
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:140)
at org.opensearch.cli.Command.main(Command.java:103)
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:139)
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:105)
For complete error details, refer to the log at /usr/share/opensearch/logs/opensearch-cluster.log

The log file /usr/share/opensearch/logs/opensearch-cluster.log is empty

To Reproduce
Steps to reproduce the behavior:

1. Run the latest helm template in kubernetes
2. View the logs in any of the nodes

Expected behavior
The Opensearch cluster should start up successfully

Plugins
Please list all plugins currently enabled.

opensearch-alerting
opensearch-anomaly-detection
opensearch-asynchronous-search
opensearch-cross-cluster-replication
opensearch-index-management
opensearch-job-scheduler
opensearch-knn
opensearch-notebooks
opensearch-performance-analyzer
opensearch-reports-scheduler
opensearch-security
opensearch-sql

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: Kubernetes
• Version: 1.18

Additional context
Add any other context about the problem here.

[anasalkouz]

@opensearch-project/transfer-request

[CEHENKLE]

moving to helm-charts. If you think that was an error, please lmk. :)

[samirmajen]

Not sure if it's helm specifically or a problem with the codebase itself

[TheAlgo]

@samirmajen Did you use the official helm chart for opensearch?

[samirmajen]

Yep

[samirmajen]

Looks like it doesn't like the env vars node.0 node.1 and node.2 which are in the official chart

[tomasodehnal]

Faced the same issue, removing the env vars (these lines from the statefulset definition) helped.

[samirmajen]

@tomasodehnal did you get it stable after that? I then get tls errors

[opensearch-cluster-master-2] OpenSearchException[Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/]

[tomasodehnal]

@samirmajen Beside the env vars I performed no other changes and fresh installation without any custom values worked. Are you using demo certs?

[erbzn]

I am also facing this problem when moving from Chart 1.2.4 to 1.3.0.
Are these changes causing problems?
Env variables change from

        - name: node.data
          value: "true"
        - name: node.ingest
          value: "true"
        - name: node.master
          value: "true"
        - name: node.remote_cluster_client
          value: "true"

to

        - name: node.0
          value: master
        - name: node.1
          value: ingest
        - name: node.2
          value: data
        - name: node.3
          value: remote_cluster_client

[samirmajen]

yes these values are the issue, as @tomasodehnal mentioned switching to a previous helm version fixes the issue. The only changes between versions are those env vars. Erbzn your values are the wrong way round, the incorrect ones are:

- name: node.0
  value: "master"
- name: node.1
  value: "ingest"
- name: node.2
  value: "data"
- name: node.3
  value: "remote_cluster_client"

These should be changed to :

- name: node.data
  value: "true"
- name: node.ingest
  value: "true"
- name: node.master
  value: "true"
- name: node.remote_cluster_client
  value: "true"

[peterzhuamazon]

Thanks guys for the investigation, would love if any of you could issue a PR for the fix.

@erbzn @samirmajen @tomasodehnal

Thanks.

[tomasodehnal]

@peterzhuamazon PR sent. The new environment variable node.roles is now set to comma separated values from the helm roles list.