Do not return usage in GET /api/stats

[afharo]

TL;DR:

We should stop returning the usage data in the GET /api/stats: Dismiss the query parameter exclude_usage, and always behave as exclude_usage=true.

Why?

In the past, we used the monitoring infrastructure to collect and send telemetry (Product Usage). This API helped with returning the Snapshot Telemetry information when requested with the query parameter ?exclude_usage=false.

However, as our product telemetry grew, the generation of the report took longer, creating issues in the frequency we poll info for monitoring (10s by default).

Metricbeat stopped polling for that info in 7.11 (it always sets exclude_usage=true), but we never removed the capability in Kibana for backwards compatibility. When analyzing support requests, we've realized there are some external services that abuse this API by calling exclude_usage=false. I think it's time we completely decouple this API (intended for monitoring) and the usage data.

Implementation details:

For the purpose of backwards compatibility, we should support the query parameter (allow it to be true/false), but Kibana would dismiss the parameter and act as if it's always true (do not extend the payload with telemetry usage):

https://github.com/elastic/kibana/blob/main/src/plugins/usage_collection/server/routes/stats/stats.ts#L88

Follow ups:

Potentially, we could also apply the changes below, but I'd rather create follow-up issues to keep the scope of this task controlled.

• Move the API to Core: this API will only return ops metrics, and it's essential for the observability of the platform, we might want to move it to core from now on.

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[nnacin]

Is this issue available to be worked on?

[afharo]

@nnacin, absolutely! Feel free to take it! PRs are always welcomed 🧡

[MohitBansal321]

maybe we need to change this default too
https://github.com/elastic/kibana/blob/main/src/plugins/usage_collection/server/routes/stats/stats.ts#L80
would you assign this issue to me

[MohitBansal321]

Let me know we need to change false to true
in these lines
https://github.com/elastic/kibana/blob/main/src/plugins/usage_collection/server/routes/stats/stats.ts#L80
https://github.com/elastic/kibana/blob/main/src/plugins/usage_collection/server/routes/stats/stats.ts#L88
what else other also

[afharo]

@MohitBansal321, it looks like @nnacin has a PR almost ready. Thank you both for contributing 🧡

[MohitBansal321]

It is okay if someone works on it that's good

[MohitBansal321]

But don't do this assign someone other and again unassign

[afharo]

Apologies! 🙏

[MohitBansal321]

Apologies! 🙏

It's okay @afharo let's connect on github