[Monitoring][Alerting] CCR read exceptions alert

[igoristic]

Resolves: #79990

The alerting query groups remote clusters with their relative follow indices. This isn't a per node type an alert nor is it a threshold type of alert. It can either be enabled or disabled.

UI/UX:

---

Testing:

1. Setup a basic CCR environment, doc link | x-pack ver

2. Confirm you have a CCR setup with a leader/follower relationship via Kibana:
   .../app/management/data/cross_cluster_replication and that the CRR monitoring metrics are working

3. I don't know a "legitimate" way of triggering the read exceptions, so I'm doing it manually, eg:

PUT .monitoring-es-7-2020.12.15/_doc/abc123
{
  "cluster_uuid": "BcK-0pmsQniyPQfZuauuXw",
  "timestamp": "2020-12-15T04:36:44.402Z",
  "interval_ms": 10000,
  "type": "ccr_stats",
  "source_node": {
    "uuid": "WJaWz2XIR8mqsZDy7eeykA",
    "host": "10.46.8.145",
    "transport_address": "10.46.8.145:19157",
    "ip": "10.46.8.145",
    "name": "instance-0000000000",
    "timestamp": "2020-12-10T07:01:52.391Z"
  },
  "ccr_stats": {
    "remote_cluster": "BcK-0pmsQniyPQfZuauuXw_remote_cluster_1",
    "leader_index": ".leader_index_1",
    "follower_index": ".follower_index_1",
    "shard_id": 1,
    "leader_global_checkpoint": 2,
    "leader_max_seq_no": 3,
    "follower_global_checkpoint": 4,
    "follower_max_seq_no": 5,
    "last_requested_seq_no": 6,
    "outstanding_read_requests": 7,
    "outstanding_write_requests": 8,
    "write_buffer_operation_count": 9,
    "write_buffer_size_in_bytes": 10,
    "follower_mapping_version": 11,
    "follower_settings_version": 12,
    "follower_aliases_version": 13,
    "total_read_time_millis": 14,
    "total_read_remote_exec_time_millis": 15,
    "read_exceptions": [
      {
        "exception": {
          "type": "read_exceptions_type_1",
          "reason": "read_exceptions_reason_1"
        }
      }
    ],
    "successful_read_requests": 1,
    "failed_read_requests": 1,
    "operations_read": 3,
    "bytes_read": 4,
    "total_write_time_millis": 5,
    "successful_write_requests": 6,
    "failed_write_requests": 7,
    "operations_written": 8,
    "time_since_last_read_millis": 9,
    "fatal_exception": {
      "type": "fatal_exception_type",
      "reason": "fatal_exception_reason"
    }
  }
}

Be sure to update the timestamp and the index name to be more recent

[elasticmachine]

Pinging @elastic/stack-monitoring (Team:Monitoring)

[chrisronline]

Great work here! Code looks solid!

A couple of things I noticed off the bat:

1. The server log needs proper escaping: Server log: CCR read exceptions alert is firing for the following remote clusters: Monitoring. Verify follower/leader index relationships across the affected remote clusters.

2. I'm not seeing any presence of the alerts on the CCR monitoring pages, even though we link to them from the next steps. I feel like we should follow the same pattern as other alerts where we surface the presence of the firing alert on the CCR listing page, as well as the individual shard pages.

3. The non-server log actions look off. Typically we have a condensed message in the context.internalShortMessage and something longer (with a deep link back to Kibana) in context.internalFullMessage

I'm going to kick it back to you to start looking into these while I dig more into the review.

[chrisronline]

Code looks great! Made a few comments

[chrisronline]

I don't know if we need to do any aggs actually.

The alert description reads:

Alert if any CCR read exceptions have been detected

I think we could get away with just searching for the past {duration} and only looking at documents that have read_exceptions (see my other comment in this file) and return that.

[chrisronline]

We'd just have to make sure we de-dupe the list, which seems like a fairly easy task (using a Set or creating a byId object and only taking the values)

[igoristic]

I don't see any benefits of doing it locally. With aggs we get "less" data back which I think makes up for its performance degradation (if there are any). Maybe it's fine since we're doing aggregation on filtered data anyways

[igoristic]

Or, maybe something I'm missing. I'm willing to explore this option, but perhaps as a post/separate task

[igoristic]

@chrisronline Thank you for the quick review!

I'm not seeing any presence of the alerts on the CCR monitoring pages

I agree this is a problem, and we have other areas in our app currently that are missing this level of granularity.

I tried addressing it here, but there's just too much involved. We basically have to make the "by nodes" logic more generic, and I'm worried about causing regressions on other listing/detail pages (with all the "unrelated" code changes). Maybe we can address these UI/UX enhancements in separate/post PRs?

...Verify follower/leader index

I fixed this by changing it to ...follower and leader, since there's no way to encode a forward slash (afaik)

[chrisronline]

I tried addressing it here, but there's just too much involved. We basically have to make the "by nodes" logic more generic, and I'm worried about causing regressions on other listing/detail pages (with all the "unrelated" code changes). Maybe we can address these UI/UX enhancements in separate/post PRs?

This confuses me a little. I'd hoped the work done in #83681 would make these sorts of things more manageable in a smaller time window, but maybe I misunderstood.

FWIW, I spent a little bit of time this morning making the changes I imagine are necessary for this and it looks like it isn't too bad. See https://gist.github.com/chrisronline/4fb0534c0d6ba803af56c42c07b2bc97

WDYT about that approach? FWIW, I think there are things to make that code a bit better but it should work for this PR.

[igoristic]

@chrisronline This is ready for another review

I think we have indeed improved our alerting development flow, but I agree there's always room for improvements.

Thank you for the example gist, it helped my out a lot! I took a slightly different approach by changing our node* terminology to a more "dumb" item/component ideology and using their respective meta keys for the relevant filters.

I have state my reasons for splitting this up, but mainly it was to also make the FF

[chrisronline]

Thank you for the example gist, it helped my out a lot! I took a slightly different approach by changing our node* terminology to a more "dumb" item/component ideology and using their respective meta keys for the relevant filters.

FWIW, in another PR, I mentioned:

Originally, the missing monitoring data alert did, as we alerted on more than one stack product. I understand your desire to revert this, but it feels a mistake to have the base alert assume the stack product for the alert. Here and here are examples of this.

This is basically what I meant. We'd run into a scenario that node* didn't match.

[igoristic]

This is basically what I meant. We'd run into a scenario that node* didn't match

Yeah, I see what you mean now, but I still don't think "products" would be the best approach here. I was thinking something like: ui.label: 'Node 001' and ui.key: 'nodeId' (would be used as the identifier to filter on). This way we can make everything generic.

I think we should reserve this discussion outside the PR though

[chrisronline]

This is looking great! Thanks for all the hard work here @igoristic!

Code looks pretty good and I think I'm done with that part of the review.

Functionally, I can see us adding one thing:

It'd be nice to somewhere list the actual exception. I think it is available in the monitoring documents and it will help the user understand what the root problem might be.

[chrisronline]

I reached out to the ES team and the easiest way to simulate an exception is to close the follower index. See this doc

Here is what I see when I do this:

Perhaps there is a better way to show this error than in a <EuiCode> block, since it's a bit more structured? Perhaps it can be baked into the original messaging somehow?

Also, we should enable setup mode on the CCR pages, as they feature alerts and users should be able to access the config without an alert firing. This is an example of supporting this

[igoristic]

@chrisronline

Perhaps there is a better way to show this error than in a block, since it's a bit more structured? Perhaps it can be baked into the original messaging somehow?

I agree the structure is simple, but I decided not to bake it into "our" description for several reasons:

• We can't localize it
• Your specific example is simple, but what if some of them get "big" and include the trace/dump in the reason

Also, I think the code style here expresses that this is something that came from the server, and not something we assumed (or made generic). I explicitly decided to add ...ui.code so we can throw traces in there and any other body of text that we know nothing about (talking about future alerts)

[chrisronline]

LGTM! Great work here!

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: b39bd69

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
monitoring	616	617	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
monitoring	959.9KB	979.5KB	+19.5KB

Distributable file count

id	before	after	diff
default	48057	48062	+5

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
monitoring	36.4KB	37.5KB	+1.2KB

Unknown metric groups

async chunk count

id	before	after	diff
monitoring	7	8	+1

History

• 💚 Build #95333 succeeded e724db1
• 💚 Build #95281 succeeded 39de03b
• 💔 Build #95066 failed 5abba03
• 💚 Build #94704 succeeded ef76de1
• 💚 Build #94343 succeeded 55a5adc

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[igoristic]

Backport:
7.x: ae4307f
7.4: cd163b2