Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Collect cloud metadata #826

Merged
merged 40 commits into from
Jun 26, 2020
Merged
Changes from 1 commit
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
7b42a02
Add cloud metadata skeleton
basepi May 14, 2020
056bf90
Add config value
basepi May 14, 2020
a62b97f
Add aws metadata collection
basepi May 14, 2020
2a18569
Use resp instead of r
basepi Jun 9, 2020
f8f32e1
Add GCP metadata
basepi Jun 9, 2020
ca820d4
Add Azure metadata
basepi Jun 9, 2020
97bf332
Refactor headers
basepi Jun 9, 2020
e101451
Use older metadata service version to guarantee availability
basepi Jun 9, 2020
dba4ad9
Add configuration docs
basepi Jun 9, 2020
f06500f
Use a single GCP metadata call with recursive=true
basepi Jun 10, 2020
4519b8f
Use urllib3 properly
basepi Jun 10, 2020
4c0a2bd
Fix some copy pasta
basepi Jun 10, 2020
4d4f0ea
Normalize variable names
basepi Jun 10, 2020
6960498
Pop zone if empty in azure
basepi Jun 11, 2020
5dab403
Remove provider guessing
basepi Jun 12, 2020
9331d45
Token step doesn't appear necessary for AWS
basepi Jun 12, 2020
7f0044c
Just kidding, token needed for IMDSv2
basepi Jun 12, 2020
dd40738
Actually import cloud
basepi Jun 16, 2020
a961180
Add a socket call to avoid urllib3 logs
basepi Jun 16, 2020
34d9ba2
Add tests
basepi Jun 16, 2020
c68b8d5
Merge remote-tracking branch 'upstream/master' into cloudmetadata
basepi Jun 16, 2020
dc96df8
Add changelog
basepi Jun 16, 2020
073562c
Don't bother with tcp handshake for google metadata
basepi Jun 17, 2020
a49b149
Add logs for missing metadata when provider is defined
basepi Jun 17, 2020
60b6f96
Move metadata generation to post-thread-start
basepi Jun 17, 2020
22ba112
Merge remote-tracking branch 'upstream/master' into cloudmetadata
basepi Jun 17, 2020
1557c91
Update tests to allow metadata to be generated in transport
basepi Jun 17, 2020
e126f44
Merge remote-tracking branch 'upstream/master' into cloudmetadata
basepi Jun 17, 2020
4116213
For GCP IDs to string
basepi Jun 17, 2020
489c5ec
Consolidate ret
basepi Jun 17, 2020
7d2765f
We don't need warning counts, just check for the warning we care about
basepi Jun 18, 2020
ef87c56
Merge branch 'master' into cloudmetadata
basepi Jun 24, 2020
13d4680
Merge branch 'master' into cloudmetadata
basepi Jun 24, 2020
0f9a048
Fix header encodings test timing
basepi Jun 24, 2020
cff2225
Merge branch 'master' into cloudmetadata
basepi Jun 24, 2020
d4e8d19
Turn off cloud metadata for tests
basepi Jun 26, 2020
3f0ec61
Merge remote-tracking branch 'upstream/master' into cloudmetadata
basepi Jun 26, 2020
1bbb274
Typo
basepi Jun 26, 2020
8d3739b
We do need a short sleep here
basepi Jun 26, 2020
f131101
Remove unused import
basepi Jun 26, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 7 additions & 6 deletions elasticapm/utils/cloud.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,15 +48,16 @@ def aws_metadata():
not found, return an empty dictionary
"""
ret = {}
basepi marked this conversation as resolved.
Show resolved Hide resolved
http = urllib3.PoolManager()

try:
ttl_header = {"X-aws-ec2-metadata-token-ttl-seconds": "300"}
token_url = "http://169.254.169.254/latest/api/token"
token_request = urllib3.request("PUT", token_url, headers=ttl_header, timeout=3.0)
basepi marked this conversation as resolved.
Show resolved Hide resolved
token_request = http.request("PUT", token_url, headers=ttl_header, timeout=3.0)
token = token_request.data.decode("utf-8")
aws_token_header = {"X-aws-ec2-metadata-token": token}
resp = json.loads(
urllib3.request(
http.request(
"GET",
"http://169.254.169.254/latest/dynamic/instance-identity/document",
headers=aws_token_header,
Expand Down Expand Up @@ -85,12 +86,13 @@ def gcp_metadata():
"""
ret = {}
headers = {"Metadata-Flavor": "Google"}
http = urllib3.PoolManager()

try:
ret["provider"] = "gcp"

metadata = json.loads(
urllib3.request(
http.request(
"GET",
"http://metadata.google.internal/computeMetadata/v1/?recursive=true",
headers=headers,
Expand All @@ -104,8 +106,6 @@ def gcp_metadata():
ret["region"] = ret["availability_zone"].rsplit("-", 1)[0]
ret["machine"] = {"type": metadata["instance"]["machineType"]}

# TODO should we use the project information for account.id and account.name?

except Exception:
# Not on a gcp box
return {}
Expand All @@ -120,12 +120,13 @@ def azure_metadata():
"""
ret = {}
headers = {"Metadata": "true"}
http = urllib3.PoolManager()

try:
# Can't use newest metadata service version, as it's not guaranteed
# to be available in all regions
resp = json.loads(
urllib3.request(
http.request(
"GET",
"http://169.254.169.254/metadata/instance/compute?api-version=2019-08-15",
headers=headers,
Expand Down