Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Infrastructure UI] Mismatch between table and KPI averages for RX and TX #152328

Closed
crespocarlos opened this issue Feb 28, 2023 · 2 comments · Fixed by #152333
Closed

[Infrastructure UI] Mismatch between table and KPI averages for RX and TX #152328

crespocarlos opened this issue Feb 28, 2023 · 2 comments · Fixed by #152333
Assignees
@crespocarlos
Labels
bug Fixes for quality problems that affect the customer experience Feature:Metrics UI Metrics UI feature Feature:ObsHosts Hosts feature within Observability Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services

Comments

@crespocarlos
Copy link
Contributor

crespocarlos commented Feb 28, 2023 •
edited
Loading

Summary

When the date range filter is narrower than 10 minutes, the averages in the KPIs are prone to differ from those of the table

Image

For the KPIs, the average is calculated using the values from the buckets, and in these situations with narrower ranges, it's likely that the query will return more bucket items with null values.

Implementation hint

  • We could filter out null items from the calculation, but since the snapshot api is used in other places, I'm not entirely sure if this would cause any negative impact.

Acceptance criteria

  • Both KPI and table values should match
@crespocarlos crespocarlos added Feature:Metrics UI Metrics UI feature Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services Feature:ObsHosts Hosts feature within Observability labels Feb 28, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

@crespocarlos crespocarlos added the bug Fixes for quality problems that affect the customer experience label Feb 28, 2023
@crespocarlos crespocarlos self-assigned this Feb 28, 2023
@neptunian
Copy link
Contributor

Similar to the issue in the dashboards that use the date histogram to calculate similar metrics: elastic/integrations#1437

Could we have another aggregation that does not use the date histogram for calculating the average and instead gets the average of that field in the time range?

@crespocarlos crespocarlos mentioned this issue Feb 28, 2023
Merged
crespocarlos added a commit that referenced this issue Mar 1, 2023
@crespocarlos @kibanamachine
[Infrastructure UI] Filter out null bucket items from average calcula…
f6a0b88 
…tion (#152333)

## Summary

Closes [#152328](#152328)

This PR fixes the average calculation in the Snapshot API, filtering out
buckets with null values from it, which are more likely to appear with
queries that use small data ranges.

The results after this change are equal to what Elasticsearch would
calculate in the avg aggregation


### How to test 

- Make sure you have metrics data (either through enabling the system
module in metricbeat or connecting your local kibana to an oblt-cli
cluster)
- Navigate to `Infrastructure` > `Hosts`
- Filter the results to see a single host
- Change the data range filter and compare the KPIs against the table.
- Validate other pages that use the Snapshot API (Inventory UI and
Metrics UI to see if the results there are still correct

---------

Co-authored-by: Kibana Machine <[email protected]>
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Mar 1, 2023
@crespocarlos
[Infrastructure UI] Filter out null bucket items from average calcula…
50865f4 
…tion (elastic#152333)

## Summary

Closes [elastic#152328](elastic#152328)

This PR fixes the average calculation in the Snapshot API, filtering out
buckets with null values from it, which are more likely to appear with
queries that use small data ranges.

The results after this change are equal to what Elasticsearch would
calculate in the avg aggregation

### How to test

- Make sure you have metrics data (either through enabling the system
module in metricbeat or connecting your local kibana to an oblt-cli
cluster)
- Navigate to `Infrastructure` > `Hosts`
- Filter the results to see a single host
- Change the data range filter and compare the KPIs against the table.
- Validate other pages that use the Snapshot API (Inventory UI and
Metrics UI to see if the results there are still correct

---------

Co-authored-by: Kibana Machine <[email protected]>
(cherry picked from commit f6a0b88)
kibanamachine added a commit that referenced this issue Mar 1, 2023
@kibanamachine @crespocarlos
[8.7] [Infrastructure UI] Filter out null bucket items from average c…
1d25c4f 
…alculation (#152333) (#152461)

# Backport

This will backport the following commits from `main` to `8.7`:
- [[Infrastructure UI] Filter out null bucket items from average
calculation (#152333)](#152333)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Carlos
Crespo","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-03-01T14:31:35Z","message":"[Infrastructure
UI] Filter out null bucket items from average calculation
(#152333)\n\n## Summary\r\n\r\nCloses
[#152328](https://github.com/elastic/kibana/issues/152328)\r\n\r\nThis
PR fixes the average calculation in the Snapshot API, filtering
out\r\nbuckets with null values from it, which are more likely to appear
with\r\nqueries that use small data ranges.\r\n\r\nThe results after
this change are equal to what Elasticsearch would\r\ncalculate in the
avg aggregation\r\n\r\n\r\n### How to test \r\n\r\n- Make sure you have
metrics data (either through enabling the system\r\nmodule in metricbeat
or connecting your local kibana to an oblt-cli\r\ncluster)\r\n- Navigate
to `Infrastructure` > `Hosts`\r\n- Filter the results to see a single
host\r\n- Change the data range filter and compare the KPIs against the
table.\r\n- Validate other pages that use the Snapshot API (Inventory UI
and\r\nMetrics UI to see if the results there are still
correct\r\n\r\n---------\r\n\r\nCo-authored-by: Kibana Machine
<[email protected]>","sha":"f6a0b886b143dea6b3b6f8101e21be5c1816558a","branchLabelMapping":{"^v8.8.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Feature:Metrics
UI","Team:Infra Monitoring
UI","release_note:skip","backport:prev-minor","v8.7.0","Feature:ObsHosts","v8.8.0"],"number":152333,"url":"https://github.com/elastic/kibana/pull/152333","mergeCommit":{"message":"[Infrastructure
UI] Filter out null bucket items from average calculation
(#152333)\n\n## Summary\r\n\r\nCloses
[#152328](https://github.com/elastic/kibana/issues/152328)\r\n\r\nThis
PR fixes the average calculation in the Snapshot API, filtering
out\r\nbuckets with null values from it, which are more likely to appear
with\r\nqueries that use small data ranges.\r\n\r\nThe results after
this change are equal to what Elasticsearch would\r\ncalculate in the
avg aggregation\r\n\r\n\r\n### How to test \r\n\r\n- Make sure you have
metrics data (either through enabling the system\r\nmodule in metricbeat
or connecting your local kibana to an oblt-cli\r\ncluster)\r\n- Navigate
to `Infrastructure` > `Hosts`\r\n- Filter the results to see a single
host\r\n- Change the data range filter and compare the KPIs against the
table.\r\n- Validate other pages that use the Snapshot API (Inventory UI
and\r\nMetrics UI to see if the results there are still
correct\r\n\r\n---------\r\n\r\nCo-authored-by: Kibana Machine
<[email protected]>","sha":"f6a0b886b143dea6b3b6f8101e21be5c1816558a"}},"sourceBranch":"main","suggestedTargetBranches":["8.7"],"targetPullRequestStates":[{"branch":"8.7","label":"v8.7.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.8.0","labelRegex":"^v8.8.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/152333","number":152333,"mergeCommit":{"message":"[Infrastructure
UI] Filter out null bucket items from average calculation
(#152333)\n\n## Summary\r\n\r\nCloses
[#152328](https://github.com/elastic/kibana/issues/152328)\r\n\r\nThis
PR fixes the average calculation in the Snapshot API, filtering
out\r\nbuckets with null values from it, which are more likely to appear
with\r\nqueries that use small data ranges.\r\n\r\nThe results after
this change are equal to what Elasticsearch would\r\ncalculate in the
avg aggregation\r\n\r\n\r\n### How to test \r\n\r\n- Make sure you have
metrics data (either through enabling the system\r\nmodule in metricbeat
or connecting your local kibana to an oblt-cli\r\ncluster)\r\n- Navigate
to `Infrastructure` > `Hosts`\r\n- Filter the results to see a single
host\r\n- Change the data range filter and compare the KPIs against the
table.\r\n- Validate other pages that use the Snapshot API (Inventory UI
and\r\nMetrics UI to see if the results there are still
correct\r\n\r\n---------\r\n\r\nCo-authored-by: Kibana Machine
<[email protected]>","sha":"f6a0b886b143dea6b3b6f8101e21be5c1816558a"}}]}]
BACKPORT-->

Co-authored-by: Carlos Crespo <[email protected]>
bmorelli25 pushed a commit to bmorelli25/kibana that referenced this issue Mar 10, 2023
@crespocarlos @kibanamachine @bmorelli25
[Infrastructure UI] Filter out null bucket items from average calcula…
b32d60b 
…tion (elastic#152333)

## Summary

Closes [elastic#152328](elastic#152328)

This PR fixes the average calculation in the Snapshot API, filtering out
buckets with null values from it, which are more likely to appear with
queries that use small data ranges.

The results after this change are equal to what Elasticsearch would
calculate in the avg aggregation


### How to test 

- Make sure you have metrics data (either through enabling the system
module in metricbeat or connecting your local kibana to an oblt-cli
cluster)
- Navigate to `Infrastructure` > `Hosts`
- Filter the results to see a single host
- Change the data range filter and compare the KPIs against the table.
- Validate other pages that use the Snapshot API (Inventory UI and
Metrics UI to see if the results there are still correct

---------

Co-authored-by: Kibana Machine <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@crespocarlos crespocarlos

Labels
bug Fixes for quality problems that affect the customer experience Feature:Metrics UI Metrics UI feature Feature:ObsHosts Hosts feature within Observability Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Infrastructure UI] Filter out null bucket items from average calculation crespocarlos/kibana
3 participants
@neptunian @crespocarlos @elasticmachine