[Infra UI] Add cloud metrics and cloud/host info to metadata endpoint #41836

simianhacker · 2019-07-23T22:20:45Z

Summary

This PR closes #39280 by adding a nodes cloud/host details to the metadata. Then using cloud.instance.id to query for cloud metrics event.dataset. For Kubernetes pods, we first have to find the kubernetes.node.name then use that to get the host's cloud/host details because the host/cloud data attached to the Kubernetes metrics are the host that reported the metric which is not always the host where the pod lives.

Checklist

Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR.

~~This was checked for cross-browser compatibility, including a check against IE11~~
~~Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support~~
~~Documentation was added for features that require explanation or tutorials~~
Unit or functional tests were updated or added to match the most common scenarios
~~This was checked for keyboard-only and screenreader accessibility~~

elasticmachine · 2019-07-24T21:18:45Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-07-24T23:19:17Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-07-25T16:35:10Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-07-25T17:12:40Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-07-25T19:48:37Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-07-26T00:20:42Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-07-26T15:55:10Z

Pinging @elastic/infra-logs-ui

elasticmachine · 2019-07-26T16:53:27Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

x-pack/legacy/plugins/infra/server/lib/adapters/metadata/elasticsearch_metadata_adapter.ts

elasticmachine · 2019-07-29T15:53:16Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

simianhacker · 2019-07-31T22:21:22Z

@skh Refactor is done, wanna take another 👀

skh · 2019-08-01T11:16:27Z

This is a way larger refactor than I was thinking about, I meant only getting rid of the separate adapter and domain parts, but 👍 👍 for cleaning away a piece of GraphQL!

skh

First, great effort to get away from GraphQL for one more endpoint! 👍

I have commented inline with a few questions.

In general, I see that we are following two different patterns in the HTTP API endpoints that are currently being added. One is everything related to logs, the other is the metric explorer endpoint and now this one. The differences are:

route and types below common or below server
using io-ts vs. not
using ui/kfetch vs our own fetch helper

This worries me a bit, because we're introducing more variety even as we are still busy cleaning up older tech debt. What do think?

I've tested this by opening the node detail page for the AWS instance in the Observability-8.x.x cluster. The response looks good:

{
    "features": [
        {
            "name": "system.process",
            "source": "metrics"
        },
        {
            "name": "system.network",
            "source": "metrics"
        },
        {
            "name": "system.cpu",
            "source": "metrics"
        },
        {
            "name": "system.load",
            "source": "metrics"
        },
        {
            "name": "system.memory",
            "source": "metrics"
        },
        {
            "name": "system.process.summary",
            "source": "metrics"
        },
        {
            "name": "system.socket.summary",
            "source": "metrics"
        },
        {
            "name": "system.filesystem",
            "source": "metrics"
        },
        {
            "name": "system.fsstat",
            "source": "metrics"
        },
        {
            "name": "system.uptime",
            "source": "metrics"
        },
        {
            "name": "system.process",
            "source": "metrics"
        },
        {
            "name": "system.network",
            "source": "metrics"
        },
        {
            "name": "system.cpu",
            "source": "metrics"
        },
        {
            "name": "system.load",
            "source": "metrics"
        },
        {
            "name": "system.memory",
            "source": "metrics"
        },
        {
            "name": "system.process.summary",
            "source": "metrics"
        },
        {
            "name": "system.socket.summary",
            "source": "metrics"
        },
        {
            "name": "system.filesystem",
            "source": "metrics"
        },
        {
            "name": "system.fsstat",
            "source": "metrics"
        },
        {
            "name": "aws.ec2",
            "source": "metrics"
        },
        {
            "name": "system.uptime",
            "source": "metrics"
        }
    ],
    "id": "ip-172-31-47-9.us-east-2.compute.internal",
    "info": {
        "cloud": {
            "account": {
                "id": "015351775590"
            },
            "availability_zone": "us-east-2c",
            "image": {
                "id": "ami-0d8f6eb4f641ef691"
            },
            "instance": {
                "id": "i-011454f72559c510b"
            },
            "machine": {
                "type": "t2.micro"
            },
            "provider": "aws",
            "region": "us-east-2"
        },
        "host": {
            "architecture": "x86_64",
            "containerized": false,
            "hostname": "ip-172-31-47-9.us-east-2.compute.internal",
            "id": "ded64cbff86f478990a3dfbb63a8d238",
            "name": "ip-172-31-47-9.us-east-2.compute.internal",
            "os": {
                "codename": "Karoo",
                "family": "redhat",
                "kernel": "4.14.123-111.109.amzn2.x86_64",
                "name": "Amazon Linux",
                "platform": "amzn",
                "version": "2"
            }
        }
    },
    "name": "ip-172-31-47-9.us-east-2.compute.internal"
}

The response for the monitoring host, however, also lists aws.ec2 as available metricset, though I believe it shouldn't:

{
    "features": [
        {
            "name": "system.process",
            "source": "metrics"
        },
        {
            "name": "system.network",
            "source": "metrics"
        },
        {
            "name": "system.cpu",
            "source": "metrics"
        },
        {
            "name": "system.load",
            "source": "metrics"
        },
        {
            "name": "system.memory",
            "source": "metrics"
        },
        {
            "name": "system.process.summary",
            "source": "metrics"
        },
        {
            "name": "system.socket.summary",
            "source": "metrics"
        },
        {
            "name": "system.filesystem",
            "source": "metrics"
        },
        {
            "name": "system.fsstat",
            "source": "metrics"
        },
        {
            "name": "aws.ec2",
            "source": "metrics"
        },
        {
            "name": "system.uptime",
            "source": "metrics"
        },
        {
            "name": "system.process",
            "source": "metrics"
        },
        {
            "name": "system.network",
            "source": "metrics"
        },
        {
            "name": "system.cpu",
            "source": "metrics"
        },
        {
            "name": "system.load",
            "source": "metrics"
        },
        {
            "name": "system.memory",
            "source": "metrics"
        },
        {
            "name": "system.process.summary",
            "source": "metrics"
        },
        {
            "name": "system.socket.summary",
            "source": "metrics"
        },
        {
            "name": "system.filesystem",
            "source": "metrics"
        },
        {
            "name": "system.fsstat",
            "source": "metrics"
        },
        {
            "name": "system.uptime",
            "source": "metrics"
        }
    ],
    "id": "skh-aws-monitoring-host",
    "info": {
        "cloud": {
            "availability_zone": "europe-west3-c",
            "instance": {
                "id": "1872939573891739009",
                "name": "skh-aws-monitoring-host"
            },
            "machine": {
                "type": "f1-micro"
            },
            "project": {
                "id": "elastic-observability"
            },
            "provider": "gcp"
        },
        "host": {
            "architecture": "x86_64",
            "containerized": false,
            "hostname": "skh-aws-monitoring-host",
            "id": "6d984a37b87a34f73f0c46063a439980",
            "name": "skh-aws-monitoring-host",
            "os": {
                "codename": "stretch",
                "family": "debian",
                "kernel": "4.9.0-9-amd64",
                "name": "Debian GNU/Linux",
                "platform": "debian",
                "version": "9 (stretch)"
            }
        }
    },
    "name": "skh-aws-monitoring-host"
}

The monitoring host is correctly reporting cloud metadata for itself, because it is also running in the cloud (on a GCP instance), but for this host there should not be any aws.ec2 metadata being reported as available, as it only sends aws.ec2 data for other hosts.

x-pack/legacy/plugins/infra/public/graphql/introspection.json

x-pack/legacy/plugins/infra/public/graphql/types.ts

skh · 2019-08-01T12:46:43Z

x-pack/legacy/plugins/infra/server/routes/metadata/index.ts

+ * you may not use this file except in compliance with the Elastic License.
+ */
+
+import { boomify } from 'boom';


This is not part of this change, but I notice that we require a different version of boom than the main Kibana package.json. Is there a reason for that (both that we need a newer one and that the version couldn't be bumped for all of Kibana)?

This was updated by @spalger via #37555, maybe he can explain?

x-pack/legacy/plugins/infra/server/routes/metadata/types.ts

x-pack/legacy/plugins/infra/public/containers/metadata/use_metadata.ts

simianhacker · 2019-08-01T14:57:13Z

@skh Looks like we need to filter out the event.dataset for aws.ec2 from getMetricMetadata and only add it in getCloudMetricsMetadata.

With regards to how we are doing the HTTP endpoints I just followed what I did for Metrics Explorer. I can move this one over to follow the conventions above. Then file a separate PR to convert Metrics Explorer so they are all consistent.

simianhacker · 2019-08-01T15:45:37Z

@skh It looks like we are using Joi to validate the incoming HTTP request in all the Logging API's as well. I prefer Joi for that part since it integrates nicely with Hapi. You can just give it a schema and if it finds a mismatch it returns a very nicely formatted error.

elasticmachine · 2019-08-01T16:28:54Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-08-01T18:27:12Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

…ient and server

elasticmachine · 2019-08-01T23:14:16Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

skh · 2019-08-02T15:02:24Z

@skh Looks like we need to filter out the event.dataset for aws.ec2 from getMetricMetadata and only add it in getCloudMetricsMetadata.

This works as expected now 👍

skh

Thanks for the changes towards better consistency throughout Logs & Infrastructure. Some of my questions were based on Felix' work in a draft PR (#42356 ) -- apologies if those sounded more like requests than questions, and thanks for considering them anyway.

…elastic#41836) * [Infra UI] Add cloud metrics and cloud/host info to metadata endpoint * Adding cloud metrics * Correcting the pod host/cloud info * Adding tests to metadata for new cloud/host info * Fixing test to include machine.type * Adding aws test data * Refactor metadata container into hook * Functionally complete * updating tests * Removing Metadata GraphQL endpoint and supporting files * Moving types under common/http_api and prefixing with Infra * adding filter for aws.ec2 dataset * move away from fetch to useHTTPRequest * Add decode function to useHTTPRequest; rename data to response; * Changing from Typescript types to IO-TS types and adding checks at client and server

- Adds APM feature to metadata - Closes elastic#42167 - Based on elastic#41836

…dpoint (#41836) (#42536) * [Infra UI] Add cloud metrics and cloud/host info to metadata endpoint (#41836) * [Infra UI] Add cloud metrics and cloud/host info to metadata endpoint * Adding cloud metrics * Correcting the pod host/cloud info * Adding tests to metadata for new cloud/host info * Fixing test to include machine.type * Adding aws test data * Refactor metadata container into hook * Functionally complete * updating tests * Removing Metadata GraphQL endpoint and supporting files * Moving types under common/http_api and prefixing with Infra * adding filter for aws.ec2 dataset * move away from fetch to useHTTPRequest * Add decode function to useHTTPRequest; rename data to response; * Changing from Typescript types to IO-TS types and adding checks at client and server * Fixing field type

…-or-edit-existing-rollup-job * 'master' of github.com:elastic/kibana: (67 commits) [TSVB] Shim new platform (elastic#39169) [Metric Vis] Shim new platform (elastic#42240) [Tag Cloud] Shim new platform (elastic#42348) Disable flaky request lib tests. Add es_ui_shared plugin to CODEOWNERS. Add disk space percentage to node listing (elastic#42145) [SIEM] Add chart interactions - update date picker after brush selection on charts (elastic#42440) Document HTTP service (elastic#42331) [Reporting] Sanitize 409 error log message (elastic#42495) [docs][skip ci] Maps read only access (elastic#35561) [x-pack/ftr] refactor types to be more accurate/consistent wit… (elastic#42407) [DOCS] Updates images and content in Dashboard docs (elastic#42500) Allow sorting on multiple columns in Discover (elastic#41918) [Infra UI][Logs UI] Fix autocomplete to use proper derived index pattern (elastic#42287) [ftr/cheerio] improve cheerio types to include test subject me… (elastic#42534) Upgraded EUI 13.0.0 -> 13.1.1 (elastic#42298) Increase max-old-space-size for builds (elastic#42218) [Infra UI] Add cloud metrics and cloud/host info to metadata endpoint (elastic#41836) [Logs UI][a11y] Announce name of column on remove column button (elastic#41695) Inspector 👉 New Platform (elastic#42164) Make alerting properly space aware (elastic#42081) ...

…s_autocomplete * 'master' of github.com:elastic/kibana: (189 commits) [TSVB] Shim new platform (elastic#39169) [Metric Vis] Shim new platform (elastic#42240) [Tag Cloud] Shim new platform (elastic#42348) Disable flaky request lib tests. Add es_ui_shared plugin to CODEOWNERS. Add disk space percentage to node listing (elastic#42145) [SIEM] Add chart interactions - update date picker after brush selection on charts (elastic#42440) Document HTTP service (elastic#42331) [Reporting] Sanitize 409 error log message (elastic#42495) [docs][skip ci] Maps read only access (elastic#35561) [x-pack/ftr] refactor types to be more accurate/consistent wit… (elastic#42407) [DOCS] Updates images and content in Dashboard docs (elastic#42500) Allow sorting on multiple columns in Discover (elastic#41918) [Infra UI][Logs UI] Fix autocomplete to use proper derived index pattern (elastic#42287) [ftr/cheerio] improve cheerio types to include test subject me… (elastic#42534) Upgraded EUI 13.0.0 -> 13.1.1 (elastic#42298) Increase max-old-space-size for builds (elastic#42218) [Infra UI] Add cloud metrics and cloud/host info to metadata endpoint (elastic#41836) [Logs UI][a11y] Announce name of column on remove column button (elastic#41695) Inspector 👉 New Platform (elastic#42164) Make alerting properly space aware (elastic#42081) ...

* [Infra UI] Add APM to Metadata Endpoint - Adds APM feature to metadata - Closes elastic#42167 - Based on elastic#41836 * migrating to new http api

* [Infra UI] Add APM to Metadata Endpoint - Adds APM feature to metadata - Closes #42167 - Based on #41836 * migrating to new http api

simianhacker added 2 commits July 23, 2019 15:15

[Infra UI] Add cloud metrics and cloud/host info to metadata endpoint

5aa4519

Adding cloud metrics

b7848ec

simianhacker added Feature:Metrics UI Metrics UI feature release_note:enhancement v7.4.0 v8.0.0 labels Jul 23, 2019

simianhacker self-assigned this Jul 23, 2019

Correcting the pod host/cloud info

7cb8831

Adding tests to metadata for new cloud/host info

afccd9a

simianhacker added 2 commits July 25, 2019 08:12

Fixing test to include machine.type

338918e

Merge branch 'master' of github.com:elastic/kibana into fixes-39280

50c9639

Merge branch 'master' of github.com:elastic/kibana into fixes-39280

dc6fcaa

Adding aws test data

87240a4

Merge branch 'master' of github.com:elastic/kibana into fixes-39280

2a3dcda

simianhacker added the Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services label Jul 26, 2019

simianhacker marked this pull request as ready for review July 26, 2019 18:29

simianhacker requested a review from a team as a code owner July 26, 2019 18:29

skh reviewed Jul 29, 2019

View reviewed changes

x-pack/legacy/plugins/infra/server/lib/adapters/metadata/elasticsearch_metadata_adapter.ts Outdated Show resolved Hide resolved

Merge branch 'master' of github.com:elastic/kibana into fixes-39280

4abae3b

simianhacker mentioned this pull request Jul 29, 2019

[Infra UI] Add APM to Metadata Endpoint #42197

Merged

1 task

skh reviewed Aug 1, 2019

View reviewed changes

simianhacker added 2 commits August 1, 2019 08:26

Moving types under common/http_api and prefixing with Infra

21371a9

adding filter for aws.ec2 dataset

0662c46

move away from fetch to useHTTPRequest

b46c035

simianhacker added 2 commits August 1, 2019 15:09

Add decode function to useHTTPRequest; rename data to response;

6cdd1ce

Changing from Typescript types to IO-TS types and adding checks at cl…

ebb3b0b

…ient and server

simianhacker requested a review from skh August 1, 2019 23:51

skh approved these changes Aug 2, 2019

View reviewed changes

simianhacker merged commit cf21757 into elastic:master Aug 2, 2019

simianhacker mentioned this pull request Aug 2, 2019

[7.x] [Infra UI] Add cloud metrics and cloud/host info to metadata endpoint (#41836) #42536

Merged

simianhacker added a commit to simianhacker/kibana that referenced this pull request Aug 2, 2019

[Infra UI] Add APM to Metadata Endpoint

2947214

- Adds APM feature to metadata - Closes elastic#42167 - Based on elastic#41836

jasonrhodes mentioned this pull request Aug 6, 2019

[APM] Improve type safety across application boundaries #42252

Closed

simianhacker added a commit that referenced this pull request Aug 14, 2019

[Infra UI] Add APM to Metadata Endpoint (#42197)

4a1f41e

* [Infra UI] Add APM to Metadata Endpoint - Adds APM feature to metadata - Closes #42167 - Based on #41836 * migrating to new http api

simianhacker added a commit that referenced this pull request Aug 14, 2019

[Infra UI] Add APM to Metadata Endpoint (#42197) (#43280)

c42c0b9

* [Infra UI] Add APM to Metadata Endpoint - Adds APM feature to metadata - Closes #42167 - Based on #41836 * migrating to new http api

simianhacker deleted the fixes-39280 branch April 17, 2024 14:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Infra UI] Add cloud metrics and cloud/host info to metadata endpoint #41836

[Infra UI] Add cloud metrics and cloud/host info to metadata endpoint #41836

simianhacker commented Jul 23, 2019 •

edited

Loading

elasticmachine commented Jul 24, 2019

elasticmachine commented Jul 24, 2019

elasticmachine commented Jul 25, 2019

elasticmachine commented Jul 25, 2019

elasticmachine commented Jul 25, 2019

elasticmachine commented Jul 26, 2019

elasticmachine commented Jul 26, 2019

elasticmachine commented Jul 26, 2019

elasticmachine commented Jul 29, 2019

simianhacker commented Jul 31, 2019

skh commented Aug 1, 2019 •

edited

Loading

skh left a comment •

edited

Loading

skh Aug 1, 2019

simianhacker Aug 1, 2019

simianhacker commented Aug 1, 2019

simianhacker commented Aug 1, 2019

elasticmachine commented Aug 1, 2019

elasticmachine commented Aug 1, 2019

elasticmachine commented Aug 1, 2019

skh commented Aug 2, 2019

skh left a comment

[Infra UI] Add cloud metrics and cloud/host info to metadata endpoint #41836

[Infra UI] Add cloud metrics and cloud/host info to metadata endpoint #41836

Conversation

simianhacker commented Jul 23, 2019 • edited Loading

Summary

Checklist

elasticmachine commented Jul 24, 2019

💚 Build Succeeded

elasticmachine commented Jul 24, 2019

💔 Build Failed

elasticmachine commented Jul 25, 2019

💔 Build Failed

elasticmachine commented Jul 25, 2019

💔 Build Failed

elasticmachine commented Jul 25, 2019

💔 Build Failed

elasticmachine commented Jul 26, 2019

💔 Build Failed

elasticmachine commented Jul 26, 2019

elasticmachine commented Jul 26, 2019

💚 Build Succeeded

elasticmachine commented Jul 29, 2019

💚 Build Succeeded

simianhacker commented Jul 31, 2019

skh commented Aug 1, 2019 • edited Loading

skh left a comment • edited Loading

Choose a reason for hiding this comment

skh Aug 1, 2019

Choose a reason for hiding this comment

simianhacker Aug 1, 2019

Choose a reason for hiding this comment

simianhacker commented Aug 1, 2019

simianhacker commented Aug 1, 2019

elasticmachine commented Aug 1, 2019

💚 Build Succeeded

elasticmachine commented Aug 1, 2019

💚 Build Succeeded

elasticmachine commented Aug 1, 2019

💚 Build Succeeded

skh commented Aug 2, 2019

skh left a comment

Choose a reason for hiding this comment

simianhacker commented Jul 23, 2019 •

edited

Loading

skh commented Aug 1, 2019 •

edited

Loading

skh left a comment •

edited

Loading