Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Collect cloud metadata #826

Merged
merged 40 commits into from
Jun 26, 2020
Merged

Collect cloud metadata #826

merged 40 commits into from
Jun 26, 2020

Conversation

basepi
Copy link
Contributor

@basepi basepi commented May 14, 2020

What does this pull request do?

Collects cloud metadata

Remaining work

  • Add config
  • Document config
  • Add AWS metadata
  • Add GCP metadata
  • Add Azure metadata
  • Add provider guessing
  • Add tests?

Related issues

Closes #822

@apmmachine
Copy link
Contributor

apmmachine commented May 21, 2020

💚 Build Succeeded

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #826 updated]

  • Start Time: 2020-06-26T21:26:19.438+0000

  • Duration: 30 min 41 sec

Test stats 🧪

Test Results
Failed 0
Passed 10542
Skipped 7953
Total 18495

@sethmlarson
Copy link
Contributor

Leaving a thought here: A lot of the JSON response value collecting is fragile to any change in the metadata API response structure. If any field on the JSON response were to change then all fields wouldn't be detected for that provider. Obviously unlikely to happen but something I thought of.

@basepi
Copy link
Contributor Author

basepi commented Jun 16, 2020

@sethmlarson Yes, that's definitely a concern. Unfortunately we don't have an easy way to do cross-cloud integration tests to test against the actual endpoints, though we may add that in the future. Luckily, these APIs tend to be very stable because they're heavily relied on.

@basepi basepi marked this pull request as ready for review June 16, 2020 21:58
@basepi
Copy link
Contributor Author

basepi commented Jun 24, 2020

@beniwohli This has now been tested against apm-server 7.8.0 and the latest nightly. Working in both places (and I do see the metadata in the nightly). Ready for final review!

Copy link
Contributor

@beniwohli beniwohli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one suggestion regarding the changes in sleep duration, otherwise LGTM

tests/transports/test_base.py Outdated Show resolved Hide resolved
@basepi basepi merged commit 18ebe4c into elastic:master Jun 26, 2020
axw added a commit to axw/apm-agent-go that referenced this pull request Sep 30, 2020
Based on the Python reference implementation:
elastic/apm-agent-python#826

By default we attempt to sniff metadata for all cloud
providers. This can be overridden by setting the
environment variable ELASTIC_APM_CLOUD_PROVIDER to one
of "none", "aws", "azure", or "gcp".

We set a short (100ms) socket connect timeout, and a
slightly longer overall timeout (1s) for fetching the
cloud metadata.

In tests we disable cloud metadata fetching by default.
There are currently no functional tests, as that would
rely on us running in a known cloud environment.
axw added a commit to axw/apm-agent-go that referenced this pull request Sep 30, 2020
Based on the Python reference implementation:
elastic/apm-agent-python#826

By default we attempt to sniff metadata for all cloud
providers. This can be overridden by setting the
environment variable ELASTIC_APM_CLOUD_PROVIDER to one
of "none", "aws", "azure", or "gcp".

We set a short (100ms) socket connect timeout, and a
slightly longer overall timeout (1s) for fetching the
cloud metadata.

In tests we disable cloud metadata fetching by default.
There are currently no functional tests, as that would
rely on us running in a known cloud environment.
axw added a commit to axw/apm-agent-go that referenced this pull request Sep 30, 2020
Based on the Python reference implementation:
elastic/apm-agent-python#826

By default we attempt to sniff metadata for all cloud
providers. This can be overridden by setting the
environment variable ELASTIC_APM_CLOUD_PROVIDER to one
of "none", "aws", "azure", or "gcp".

We set a short (100ms) socket connect timeout, and a
slightly longer overall timeout (1s) for fetching the
cloud metadata.

In tests we disable cloud metadata fetching by default.
There are currently no functional tests, as that would
rely on us running in a known cloud environment.
axw added a commit to axw/apm-agent-go that referenced this pull request Sep 30, 2020
Based on the Python reference implementation:
elastic/apm-agent-python#826

By default we attempt to sniff metadata for all cloud
providers. This can be overridden by setting the
environment variable ELASTIC_APM_CLOUD_PROVIDER to one
of "none", "aws", "azure", or "gcp".

We set a short (100ms) socket connect timeout, and a
slightly longer overall timeout (1s) for fetching the
cloud metadata.

In tests we disable cloud metadata fetching by default.
There are currently no functional tests, as that would
rely on us running in a known cloud environment.
axw added a commit to axw/apm-agent-go that referenced this pull request Sep 30, 2020
Based on the Python reference implementation:
elastic/apm-agent-python#826

By default we attempt to sniff metadata for all cloud
providers. This can be overridden by setting the
environment variable ELASTIC_APM_CLOUD_PROVIDER to one
of "none", "aws", "azure", or "gcp".

We set a short (100ms) socket connect timeout, and a
slightly longer overall timeout (1s) for fetching the
cloud metadata.

In tests we disable cloud metadata fetching by default.
There are currently no functional tests, as that would
rely on us running in a known cloud environment.
axw added a commit to axw/apm-agent-go that referenced this pull request Sep 30, 2020
Based on the Python reference implementation:
elastic/apm-agent-python#826

By default we attempt to sniff metadata for all cloud
providers. This can be overridden by setting the
environment variable ELASTIC_APM_CLOUD_PROVIDER to one
of "none", "aws", "azure", or "gcp".

We set a short (100ms) socket connect timeout, and a
slightly longer overall timeout (1s) for fetching the
cloud metadata.

In tests we disable cloud metadata fetching by default.
There are currently no functional tests, as that would
rely on us running in a known cloud environment.
axw added a commit to axw/apm-agent-go that referenced this pull request Sep 30, 2020
Based on the Python reference implementation:
elastic/apm-agent-python#826

By default we attempt to sniff metadata for all cloud
providers. This can be overridden by setting the
environment variable ELASTIC_APM_CLOUD_PROVIDER to one
of "none", "aws", "azure", or "gcp".

We set a short (100ms) socket connect timeout, and a
slightly longer overall timeout (1s) for fetching the
cloud metadata.

In tests we disable cloud metadata fetching by default.
There are currently no functional tests, as that would
rely on us running in a known cloud environment.
axw added a commit to axw/apm-agent-go that referenced this pull request Sep 30, 2020
Based on the Python reference implementation:
elastic/apm-agent-python#826

By default we attempt to sniff metadata for all cloud
providers. This can be overridden by setting the
environment variable ELASTIC_APM_CLOUD_PROVIDER to one
of "none", "aws", "azure", or "gcp".

We set a short (100ms) socket connect timeout, and a
slightly longer overall timeout (1s) for fetching the
cloud metadata.

In tests we disable cloud metadata fetching by default.
There are currently no functional tests, as that would
rely on us running in a known cloud environment.
axw added a commit to axw/apm-agent-go that referenced this pull request Sep 30, 2020
Based on the Python reference implementation:
elastic/apm-agent-python#826

By default we attempt to sniff metadata for all cloud
providers. This can be overridden by setting the
environment variable ELASTIC_APM_CLOUD_PROVIDER to one
of "none", "aws", "azure", or "gcp".

We set a short (100ms) socket connect timeout, and a
slightly longer overall timeout (1s) for fetching the
cloud metadata.

In tests we disable cloud metadata fetching by default.
There are currently no functional tests, as that would
rely on us running in a known cloud environment.
axw added a commit to axw/apm-agent-go that referenced this pull request Oct 1, 2020
Based on the Python reference implementation:
elastic/apm-agent-python#826

By default we attempt to sniff metadata for all cloud
providers. This can be overridden by setting the
environment variable ELASTIC_APM_CLOUD_PROVIDER to one
of "none", "aws", "azure", or "gcp".

We set a short (100ms) socket connect timeout, and a
slightly longer overall timeout (1s) for fetching the
cloud metadata.

In tests we disable cloud metadata fetching by default.
There are currently no functional tests, as that would
rely on us running in a known cloud environment.
axw added a commit to axw/apm-agent-go that referenced this pull request Oct 1, 2020
Based on the Python reference implementation:
elastic/apm-agent-python#826

By default we attempt to sniff metadata for all cloud
providers. This can be overridden by setting the
environment variable ELASTIC_APM_CLOUD_PROVIDER to one
of "none", "aws", "azure", or "gcp".

We set a short (100ms) socket connect timeout, and a
slightly longer overall timeout (1s) for fetching the
cloud metadata.

In tests we disable cloud metadata fetching by default.
There are currently no functional tests, as that would
rely on us running in a known cloud environment.
axw added a commit to elastic/apm-agent-go that referenced this pull request Oct 2, 2020
* Fetch cloud metadata

Based on the Python reference implementation:
elastic/apm-agent-python#826

By default we attempt to sniff metadata for all cloud
providers. This can be overridden by setting the
environment variable ELASTIC_APM_CLOUD_PROVIDER to one
of "none", "aws", "azure", or "gcp".

We set a short (100ms) socket connect timeout, and a
slightly longer overall timeout (1s) for fetching the
cloud metadata.

In tests we disable cloud metadata fetching by default.
There are currently no functional tests, as that would
rely on us running in a known cloud environment.
romulorosa pushed a commit to romulorosa/apm-agent-python that referenced this pull request Oct 15, 2020
* Add cloud metadata skeleton

* Add config value

* Add aws metadata collection

* Use resp instead of r

* Add GCP metadata

* Add Azure metadata

* Refactor headers

* Use older metadata service version to guarantee availability

* Add configuration docs

* Use a single GCP metadata call with recursive=true

* Use urllib3 properly

* Fix some copy pasta

* Normalize variable names

* Pop zone if empty in azure

* Remove provider guessing

* Token step doesn't appear necessary for AWS

* Just kidding, token needed for IMDSv2

* Actually import cloud

* Add a socket call to avoid urllib3 logs

* Add tests

* Add changelog

* Don't bother with tcp handshake for google metadata

* Add logs for missing metadata when provider is defined

* Move metadata generation to post-thread-start

* Update tests to allow metadata to be generated in transport

* For GCP IDs to string

* Consolidate ret

* We don't need warning counts, just check for the warning we care about

* Fix header encodings test timing

* Turn off cloud metadata for tests

* Typo

* We do need a short sleep here

* Remove unused import
romulorosa pushed a commit to romulorosa/apm-agent-python that referenced this pull request Oct 15, 2020
* Add cloud metadata skeleton

* Add config value

* Add aws metadata collection

* Use resp instead of r

* Add GCP metadata

* Add Azure metadata

* Refactor headers

* Use older metadata service version to guarantee availability

* Add configuration docs

* Use a single GCP metadata call with recursive=true

* Use urllib3 properly

* Fix some copy pasta

* Normalize variable names

* Pop zone if empty in azure

* Remove provider guessing

* Token step doesn't appear necessary for AWS

* Just kidding, token needed for IMDSv2

* Actually import cloud

* Add a socket call to avoid urllib3 logs

* Add tests

* Add changelog

* Don't bother with tcp handshake for google metadata

* Add logs for missing metadata when provider is defined

* Move metadata generation to post-thread-start

* Update tests to allow metadata to be generated in transport

* For GCP IDs to string

* Consolidate ret

* We don't need warning counts, just check for the warning we care about

* Fix header encodings test timing

* Turn off cloud metadata for tests

* Typo

* We do need a short sleep here

* Remove unused import
beniwohli pushed a commit to beniwohli/apm-agent-python that referenced this pull request Sep 14, 2021
* Add cloud metadata skeleton

* Add config value

* Add aws metadata collection

* Use resp instead of r

* Add GCP metadata

* Add Azure metadata

* Refactor headers

* Use older metadata service version to guarantee availability

* Add configuration docs

* Use a single GCP metadata call with recursive=true

* Use urllib3 properly

* Fix some copy pasta

* Normalize variable names

* Pop zone if empty in azure

* Remove provider guessing

* Token step doesn't appear necessary for AWS

* Just kidding, token needed for IMDSv2

* Actually import cloud

* Add a socket call to avoid urllib3 logs

* Add tests

* Add changelog

* Don't bother with tcp handshake for google metadata

* Add logs for missing metadata when provider is defined

* Move metadata generation to post-thread-start

* Update tests to allow metadata to be generated in transport

* For GCP IDs to string

* Consolidate ret

* We don't need warning counts, just check for the warning we care about

* Fix header encodings test timing

* Turn off cloud metadata for tests

* Typo

* We do need a short sleep here

* Remove unused import
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature] Collect cloud metadata
4 participants