Reduce severity of log messages when hitting disk watermarks #48038
Assignees
Labels
:Distributed Coordination/Allocation
All issues relating to the decision making around placing a shard (both master logic & on the nodes)
>enhancement
Comments
DaveCTurner
added
help wanted
|
Pinging @elastic/es-distributed (:Distributed/Allocation) |
DaveCTurner
added a commit
to DaveCTurner/elasticsearch
that referenced
this issue
Oct 16, 2019
Today if an Elasticsearch node reaches a disk watermark then it will repeatedly emit logging about it, which implies that some action needs to be taken by the administrator. This is misleading. Elasticsearch strives to keep nodes under the high watermark, but it is normal to have a few nodes occasionally exceed this level. Nodes may be over the low watermark for an extended period without any ill effects. This commit enhances the logging emitted by the `DiskThresholdMonitor` to be less misleading. The expected case of hitting the high watermark and immediately relocating one or more shards that to bring the node back under the watermark again is reduced in severity to `INFO`. Additionally, `INFO` messages are not emitted repeatedly. Fixes elastic#48038
DaveCTurner
added a commit
that referenced
this issue
Oct 18, 2019
Today if an Elasticsearch node reaches a disk watermark then it will repeatedly emit logging about it, which implies that some action needs to be taken by the administrator. This is misleading. Elasticsearch strives to keep nodes under the high watermark, but it is normal to have a few nodes occasionally exceed this level. Nodes may be over the low watermark for an extended period without any ill effects. This commit enhances the logging emitted by the `DiskThresholdMonitor` to be less misleading. The expected case of hitting the high watermark and immediately relocating one or more shards that to bring the node back under the watermark again is reduced in severity to `INFO`. Additionally, `INFO` messages are not emitted repeatedly. Fixes #48038
DaveCTurner
added a commit
that referenced
this issue
Oct 18, 2019
Today if an Elasticsearch node reaches a disk watermark then it will repeatedly emit logging about it, which implies that some action needs to be taken by the administrator. This is misleading. Elasticsearch strives to keep nodes under the high watermark, but it is normal to have a few nodes occasionally exceed this level. Nodes may be over the low watermark for an extended period without any ill effects. This commit enhances the logging emitted by the `DiskThresholdMonitor` to be less misleading. The expected case of hitting the high watermark and immediately relocating one or more shards that to bring the node back under the watermark again is reduced in severity to `INFO`. Additionally, `INFO` messages are not emitted repeatedly. Fixes #48038
This was referenced Feb 3, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Today if an Elasticsearch node reaches a disk watermark then it will repeatedly emit logging about it, which implies that some action needs to be taken by the administrator. This is misleading. Elasticsearch strives to keep nodes under the
highwatermark, but it is normal to have a few nodes occasionally exceed this level. Nodes may be over thelowwatermark for an extended period without any ill effects.On 2019-08-28 the distributed team discussed this and decided to reduce the severity of these messages. Messages about the
lowwatermark should not be warnings at all, and ideally should not be emitted repeatedly since we expect nodes to be over this watermark for an extended period. Messages about thehighwatermark could reasonably be warnings iff the cluster cannot relocate shards to bring nodes under the watermark again; if the cluster is actively relocating shards that will balance the disk usage then there is no need for a warning.The text was updated successfully, but these errors were encountered: