[Stack Monitoring] user stories for migration to Agent

[neptunian]

Once packages are working and fully supported within the Stack Monitoring UI, we need to create user stories for migrating to Agent and decide which, if not all, users it makes sense to encourage to do so. Then I think we should work with design to come up with how we want to express this in the UI.

To get an idea of how we currently handle UI indicators for encouraging users to migrate to metricbeat, see screenshots in the comments below. This is only for on-prem where we show these.

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[smith]

Closing this. If it's something to do we should open issues addressing the use cases.

[matschaffer]

@klacabane or @neptunian is this definitely ready to get picked up? From looking at https://github.com/elastic/integrations/blob/main/packages/elasticsearch/manifest.yml#L16 it seems like I probably can't install the existing packages on main today.

Thinking maybe the parent epic might need an issue for that.

Seems like part of the ask is UX-centric here too.

Like do we want to nudge users away from standalone metricbeat onto agent? Not sure if I do, but I know @andresrc has mentioned "migration to agent" as a goal.

So maybe we want something in the current SM UI that highlights both internal & metricbeat and recommends migrating to agent + integration?

[klacabane]

The actual testing is too early to pick up since it has dependencies on working integrations packages but for the user experience question, I'd be inclined to implement a simple detection mechanism that verifies whether we are reading any documents from metrics-*. If not, we display a callout that explains the migration to agent is possible with links to relevant integrations

btw I've documented rough steps to work with the existing packages here

[neptunian]

Like do we want to nudge users away from standalone metricbeat onto agent? Not sure if I do, but I know @andresrc has mentioned "migration to agent" as a goal.

@matschaffer Good question. We need to clarify whether we want to only encourage users using self monitoring to migrate to Agent or self monitoring AND metricbeat users.

As @klacabane said, detect collection type, or whether the user is using metrics-* or not. Though I was thinking we can modify the current getCollectionStatus, perhaps changing some variable names or adding isAgentMigrated. This is how the UI currently figures out which badges to show. This is also detects partial migration in the case where a user is using metricbeat but still has self monitoring turned on.

Currently in the UI we use colors to express monitoring collection type and migration status. green: fully migrated, yellow: partially migrated, red: migration encouraged or using "legacy" monitoring

Here are the scenarios I can think of:

a user not yet monitoring

the user would be encouraged to use Agent

user is only using self monitoring

replace all instances where we encourage the user to migrate to metricbeat to migrate to agent.

user is only using metricbeat (if we decide to encourage mb users to migrate to Agent).

instead of showing the green badge with text "Monitored by metricbeat", we show an orange badge encouraging user to migrate to Agent (same as above in the self monitoring state)

Current UI when using metricbeat:

user is in a state of partial migration between self monitoring and metricbeat

continue to encourage them to finish the metricbeat migration

user is in a state of partial migration between self monitoring and Agent

same experience as above but switch "metricbeat" to "agent"

user is in a state of partial migration between Metricbeat and Agent

same as above but replace "self monitoring" with "metricbeat". this scenario could happen whether or not we encourage MB users in the UI to migrate to Agent so we should account for it.

user has fully migrated to Agent

[miltonhultgren]

Maybe this is a curveball but as part of this effort, could we hide the text for self monitoring? :D
I know we have it there to make it as easy as possible to get started but shouldn't we drive that ease of use towards improving Agent instead?

[matschaffer]

You mean the "or set up with self monitoring"? Doesn't seem like a terrible thing to remove at this stage.

[klacabane]

I'm not sure we should completely discard self-monitoring when deprecation is still not on the horizon. For cloud the option is already disabled, for on-prem it is the fastest way to get data and a functioning UI for a quick evaluation of the product.
What about keeping the option in a documentation page ? We can have the same "No monitoring" page with Agent setup primary cta and instead of the self monitoring link have an "Explore other ways to monitor your stack" that links to this documentation page

[matschaffer]

I'm gonna put this down in favor of elastic/integrations#3849

Not sure how folks are making decent headway with integrations. Maybe Japan's internet has really just gotten that bad, but it takes at least 30min for me to finish any elastic-package operation.

[klacabane]

this is my worst case scenario (nothing cached)
( elastic-package stack up -v -d; ) 7.80s user 1.48s system 6% cpu 2:33.14 total

and this is cached scenario
( elastic-package stack up -v -d; ) 1.88s user 0.57s system 49% cpu 4.984 total

[smith]

Closing this. If it's something to do we should open issues addressing the use cases.