[APM] Remove rate aggregations

[cauemarcondes]

closes #112240

• Remove rate aggs and calculate throughput manually.
• Add API tests across different APIs comparing throughput value

[shahzad31]

Uptime changes LGTM !!

[weltenwort]

I'm having a hard time following why these changes are necessary for the logs and metrics fetchers. Could you please help me understand why we need both bucketSize and intervalString and why we switch around their meanings?

[cauemarcondes]

On the APM api I need both bucketSize and intervalString now, before we were using bucketSize as string, which is incorrect, based on the function we use to calculate the bucket size getBucketSize. Now the callback function receives both options, and for you is just a matter of using the intervalString which will have the same value as the old bucketSize.

[weltenwort]

Ok, but as implementers of the a data fetcher it would be good to know what these parameters mean. Are they redundant so we can choose which to use? Probably not, because then we wouldn't need both. But which to use when?

Maybe we could add that info to the parameter type definition?

[cauemarcondes]

They are not redundant, although they will have the bucket as string and the bucket as number (seconds) for example if intervalString is 60s buckeSize will be 60.

But which to use when?

You'll use intervalString on DateHistogram aggregation, and buckeSize if you want to calculate the rate of a bucket.

Maybe we could add that info to the parameter type definition?

I'll add a description there.

[weltenwort]

thanks for the explanation!

[sorenlouv]

Any reason why callApi doesn't return body?

Suggested change

	const throughputResponse = await callApi({ query: { environment: 'production' } });
	throughput = throughputResponse.body;
	throughput = await callApi({ query: { environment: 'production' } });

[cauemarcondes]

yes, I check the status of the request on this test https://github.com/cauemarcondes/kibana/blob/apm-removing-rate-aggs/x-pack/test/apm_api_integration/tests/services/throughput.ts#L65

[sorenlouv]

Is it necessary to filter values out?

Suggested change

	const previousPeriodMean = meanBy(
	throughputResponse.previousPeriod.filter(
	(item) => isFiniteNumber(item.y) && item.y > 0
	),
	'y'
	);
	const previousPeriodMean = meanBy(throughputResponse.previousPeriod, 'y');

[cauemarcondes]

If I don't filter values out I won't be able to compare it with the RATE variables, monthly because the interval used in the dateHistogram to fetch the comparisons is 1s and since we generated data every 1m it means that for every bucket that containes value we'll have 59 others with 0. That's why I though of filtering to only return what has value and get the mean of that.

[cauemarcondes]

@elasticmachine merge upstream

[sorenlouv]

The description for the two should be identical and just highlight the the type is different:

Suggested change

	// Size of the bucket in seconds calculated based on a time range. e.g. 60
	bucketSize: number;
	// String representing the interval in seconds that is calculated based on a time range. e.g. '60s'
	intervalString: string;
	// Bucket size in seconds (number)
	bucketSize: number;
	// Bucket size in seconds (string)
	intervalString: string;

Ideally the two was called the same even:

Suggested change

	// Size of the bucket in seconds calculated based on a time range. e.g. 60
	bucketSize: number;
	// String representing the interval in seconds that is calculated based on a time range. e.g. '60s'
	intervalString: string;
	// Bucket size in seconds (number)
	bucketSize: number;
	// Bucket size in seconds (string)
	bucketSizeString: string;

(but that is a larger refactor and probably not worth doing right now)

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: e21ffc3
• Storybooks Preview

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
observability	257	258	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
apm	2.7MB	2.7MB	-391.0B
observability	363.1KB	363.1KB	-70.0B
uptime	564.7KB	564.7KB	+4.0B
total			-457.0B

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
apm	37	36	-1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
infra	90.2KB	90.2KB	+8.0B

Unknown metric groups

API count

id	before	after	diff
observability	258	259	+1

History

• 💚 Build #160901 succeeded c11b2d5
• 💚 Build #160668 succeeded ef0c4b7
• 💚 Build #160533 succeeded eede4cd
• 💔 Build #160193 failed 8d1c743
• 💔 Build #160044 failed 4548819

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[weltenwort]

infra changes LGTM and they seems to be producing the same results

thank you!

[kibanamachine]

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.