[BUG] Integration Test Failures after OpenSearch node reaches high watermark

[cwperks]

On a few PRs I see integration test failures. The failing integration test suite is saml_auth.test.ts

You can run this suite locally with: yarn test:jest_server -- saml_auth.test.ts

Example: https://github.com/cwperks/security-dashboards-plugin/actions/runs/6509864918/job/17682323837

Relevant error messages

 ● start OpenSearch Dashboards server › Login to app/opensearch_dashboards_overview#/ when SAML is enabled

    Response Error: 403 Forbidden

      112 |     });
      113 |     console.log('Starting to Download Flights Sample Data');
    > 114 |     await wreck.post('http://localhost:5601/api/sample_data/flights', {
          |     ^
      115 |       payload: {},
      116 |       rejectUnauthorized: false,
      117 |       headers: {

and

      at SecurityClient.getSamlHeader (plugins/security-dashboards-plugin/server/backend/opensearch_security_client.ts:214:17)

  console.log
    Error: failed parsing SAML config
        at SecurityClient.getSamlHeader (/home/runner/work/security-dashboards-plugin/security-dashboards-plugin/OpenSearch-Dashboards/plugins/security-dashboards-plugin/server/backend/opensearch_security_client.ts:212:15)
        at processTicksAndRejections (node:internal/process/task_queues:95:5)
        at /home/runner/work/security-dashboards-plugin/security-dashboards-plugin/OpenSearch-Dashboards/plugins/security-dashboards-plugin/server/auth/types/saml/routes.ts:78:30
        at Router.handle (/home/runner/work/security-dashboards-plugin/security-dashboards-plugin/OpenSearch-Dashboards/src/core/server/http/router/router.ts:286:44)
        at handler (/home/runner/work/security-dashboards-plugin/security-dashboards-plugin/OpenSearch-Dashboards/src/core/server/http/router/router.ts:241:11)
        at Object.<anonymous>.exports.Manager.execute (/home/runner/work/security-dashboards-plugin/security-dashboards-plugin/OpenSearch-Dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
        at Object.<anonymous>.internals.handler (/home/runner/work/security-dashboards-plugin/security-dashboards-plugin/OpenSearch-Dashboards/node_modules/@hapi/hapi/lib/handler.js:46:20)
        at Object.<anonymous>.exports.execute (/home/runner/work/security-dashboards-plugin/security-dashboards-plugin/OpenSearch-Dashboards/node_modules/@hapi/hapi/lib/handler.js:31:20)
        at Request._lifecycle (/home/runner/work/security-dashboards-plugin/security-dashboards-plugin/OpenSearch-Dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
        at Request._execute (/home/runner/work/security-dashboards-plugin/security-dashboards-plugin/OpenSearch-

[cwperks]

Closing this issue, I was running the integ tests incorrectly.

Integ tests must be run with demo security config + plugins.security.unsupported.restapi.allow_securityconfig_modification: true in opensearch.yml

[cwperks]

Re-opening this issue. The integ tests on main are consistently failing.

Its odd that its getting a 403 trying to add sample data because its supplying the correct credentials.

I was able to reproduce the same 403 on an EC2 instance after the high watermark was exceeded.

[2023-10-16T20:22:19,349][WARN ][o.o.c.r.a.DiskThresholdMonitor] [smoketestnode] Putting index create block on cluster as all nodes are breaching high disk watermark. Number of nodes above high watermark: 1.
[2023-10-16T20:22:19,352][WARN ][o.o.c.r.a.DiskThresholdMonitor] [smoketestnode] high disk watermark [90%] exceeded on [T0-F6uKuQuGuvwNsZ01Osw][smoketestnode][/home/ubuntu/opensearch/opensearch-3.0.0-SNAPSHOT/data/nodes/0] free: 4.7gb[9.7%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete

[cwperks]

Closing this issue since #1613 has been merged.