IllegalArgumentException from node stats

[s-nel]

Elasticsearch version (bin/elasticsearch --version): 7.7.0

Plugins installed: [repository-azure, repository-s3]

JVM version (java -version): ESS

OS version (uname -a if on a Unix-like system): ESS Azure

Description of the problem including expected versus actual behavior:

Executing GET /_nodes/stats?level=shards responds with

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "Values less than -1 bytes are not supported: -8192b"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "Values less than -1 bytes are not supported: -8192b",
    "suppressed": [
      {
        "type": "illegal_state_exception",
        "reason": "Failed to close the XContentBuilder",
        "caused_by": {
          "type": "i_o_exception",
          "reason": "Unclosed object or array found"
        }
      }
    ]
  },
  "status": 400
}

Expect it to return 200 and node stats. Restarting the node fixes the issue. This is causing plan failures on ESS.

I believe #48380 is related, but this appears to be memory, not disk.

Steps to reproduce:

I don't have steps to reproduce.

Provide logs (if relevant):

Couldn't find any relevant logs from ES.

[tvernum]

I wonder if this is related to #54415

We've only seen this in CI on Debian8, but there's nothing about the analysis that implies it would be limited to that platform.

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Stats)

[danhermann]

Fixed in #56435

[henrikno]

I have a case which is very similar (Values less than -1 bytes are not supported: -1282646016b), however the stack trace shows it's during getting swap (org.elasticsearch.monitor.os.OsStats$Swap.getFree) instead of Mem.getUsed.
Also on 7.7.0 (openjdk version "14" 2020-03-17)
It looks like the fix only applied to Mem.getUsed, but probably needs to be applied here too.

Full stack trace here:
https://gist.github.com/henrikno/1fc3a2f64b12e0c50b8e51e04a16d16f

Swap looks sane in /proc/meminfo

SwapTotal:      10485756 kB
SwapFree:        2495052 kB

[danhermann]

@henrikno, thanks for the report. We'll look into that case as well. We know that some JVMs can return OS memory as zero under certain circumstances, so if we can establish that that can happen for swap as well, we can apply the same fix.

[andrew-moldovan]

As an addition to what @henrikno wrote, "restarting" the affected node doesn't seem to have fixed anything in this case, unlike the original bug report.

[danhermann]

I've opened #57312 for the negative values on the swap stats.