[Stack Monitoring] Adapt UI to processes metrics from kibana 8.0

[jasonrhodes]

Based on the work being done with the process field, we need to make sure the Stack Monitoring UI will continue to work as expected in 8.0.0 once these breaking changes are complete.

AC:

• Adapt the UI to read from both process and processes[0] for the process metric charts in the SM UI

Blocked by data generation: #117615 or elastic/beats#27569

Related:

• [Breaking][core.metrics] Remove process field from ops metrics & /stats API #104124
• RFC

[elasticmachine]

Pinging @elastic/stack-monitoring (Team:Monitoring)

[ravikesarwani]

@jasonrhodes Do we know how will this work from a backward compatibility perspective?
Will a 8.0 monitoring cluster be able to monitor 8.0 and 7.x production clusters? Will the SM UI be coded smartly to handle both older and newer kinds of documents.
If the beats API has breaking change I am assuming that means for 8.0 production cluster we will need to use a compatible 8.0 beats to collect the data.

Backwards compatibility I think will be critical for proactive monitoring project otherwise we will be forced to setup 7.x and 8.x monitoring cluster.
cc @Kushmaro

[matschaffer]

So from @mshustov 's comment in #104124 (comment) the main field is changing from process: ProcessInfo; to processes: ProcessInfo[];

At very least we'll need a beats update to upgrade a single process to a collection. I don't know what/who handles this for internal collection (or if we even have internal collection for kibana).

For the beats side my goto is @sayden - hopefully he can advise on the required beats adjustments.

We'll need to adapt the UI as well to be ready for multiple process metrics coming in per kibana node.

[matschaffer]

Started poking around this yesterday. EOD update for today:

Got metricbeat running locally and pointed at a standard yarn start kibana.

The first curiosity is that the data gets written into metricbeat-* which I thought shouldn't ever happen. Maybe it's an 8.0 only thing?

With it writing there, stack monitoring won't see the kibana nodes apparently but I can see some of the ES data:

Right now in kibana master it looks like we get the first entry of processes as the process key. #115200 will remove that key.

❯ curl -s -u elastic:changeme localhost:5601/ubd/api/stats | jq '.processes[0].memory.heap.used_bytes, .process.memory.heap.used_bytes'
562406712
562406712

I can pull a lens query of the same data, but there's no processes key available in the index.

So I think the first thing we'll need to address is https://github.com/elastic/beats/blob/master/metricbeat/module/kibana/stats/_meta/fields.yml#L88 so that metricbeat can pick up processes and index all the data.

Once we do that we'll have to update the UI.

One thing to consider here is that we should probably account for some mixing of data shapes. For example, a customer might upgrade kibana ahead of beats.

We should be clear on the failure mode in this case.

Also we might have some mixing of kibana versions represented in the stack monitoring data. For example a customer might want to show their kibana data over a 30 day time frame that includes the upgrade period.

In this case we'd have data before the upgrade with kibana.stats.process and after with kibana.stats.processes.

We'll likely want to be able to plot both data shapes continuously over the range. I'm guessing the UI will need to include plots for both fields in order to do this.

[matschaffer]

EOD report: Kind of a slow one for me today among catch up with team efforts/SDH and a sick kid, but I did start on an extension to stats_test.go to work on getting metricbeat to handle the new format (in addition to the old format).

It looks like we can't use https://github.com/elastic/beats/blob/master/metricbeat/mb/testing/testdata.go#L443 due to the extra call to /api/status, but I can at least re-use the pattern.

Once metricbeat can ingest & publish both formats I can start figuring out what to do about plotting it in SM UI as well.

Like with all field migrations, I'm kind of scratching my head about what to do about the transition period. Maybe there's something we can do with runtime fields to present older data with just process as processes:[] so the UI can just plot that.

@jasonrhodes if you're familiar with any good options we have from your transition experiences so far, let me know.

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[matschaffer]

@jasonrhodes I guess I should have read the "Ac" more carefully. I've identified the changes (process: {} to processes: [{}].

I've adapted this issue to handle adding graphing but maybe I should close this one and open a new one to address graphing. Let me know what you think.

[jasonrhodes]

@matschaffer just confirming: the work needed to adapt to the original breaking change is done elsewhere, and this ticket now reflects our plans to adopt the "processes" view in graphs (but isn't itself a breaking change), is that right?

[matschaffer]

Not sure which change you're referring to with "the original breaking change". process hasn't been removed yet, that's #115200. So in practice there hasn't been any breaking change yet.

What has happened is that processes is now available and process has been set to processes[0] for backward compatibility.

This issue covers switching graphs to use processes[] in a way that accounts for more than one process (though there's only one today).

Hope that helps clarify things :)

[matschaffer]

Oh yay! I can pick this back up before addressing #117615 - internal kibana collection includes both process and processes so I can try to work out how to plot the new one when it's present.

[lukeelmers]

"the original breaking change". process hasn't been removed yet, that's #115200. So in practice there hasn't been any breaking change yet.

This is correct, we were planning to make the breaking change, but were blocked by the failing monitoring tests, so we decided we were okay with not making the break after all (Stack FF for 8.0 is already passed).

[matschaffer]

Great to hear that @lukeelmers - in my investigation last week I think things may be more complex given kibana's lack of nested visualization support. I have a sync today with @Bamieh to discuss it.

[matschaffer]

Closing this in favor of #119560 and probably also closing #104124

If we can keep process around as an aggregate, nothing should have to change in stack monitoring today. Once we have more kibana node processes and have an idea of how we want to display them in stack monitoring UI we can take the next step of doing something (I'm in favor of multiple documents) with the process: [] data.