fetch cloud metada - common logic #1453

videnkz · 2020-10-24T21:16:28Z

refer #1264
Note: we may have problems if a new version is used in the cloud with old versions of apm-server
Note2: I didn't add cloud.account.name field in serialization, because this field value is not available in the selected providers (aws, gcp, azure).

apmmachine · 2020-10-24T21:21:04Z

❕ Build Aborted

The PR is not allowed to run in the CI yet

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Branch indexing
Reason: The PR is not allowed to run in the CI yet
Start Time: 2021-01-12T15:33:56.048+0000
Duration: 3 min 50 sec
Commit: 717c34c

Steps errors

Expand to view the steps failures

`Error signal`

Took 0 min 0 sec . View more details on here
Description: githubPrCheckApproved: The PR is not allowed to run in the CI yet. (Only users with write permission

Log output

Expand to view the last 100 lines of log output

[2021-01-12T15:36:18.642Z] Using reference repository: /var/lib/jenkins/.git-references/apm-agent-java.git
[2021-01-12T15:36:18.649Z] Fetching upstream changes from https://github.com/elastic/apm-agent-java.git
[2021-01-12T15:36:18.649Z]  > git --version # timeout=10
[2021-01-12T15:36:18.666Z]  > git --version # 'git version 2.17.1'
[2021-01-12T15:36:18.666Z] using GIT_ASKPASS to set credentials GitHub user @apmmachine User + Personal Access Token
[2021-01-12T15:36:18.668Z]  > git fetch --no-tags --progress -- https://github.com/elastic/apm-agent-java.git +refs/heads/*:refs/remotes/origin/* # timeout=15
[2021-01-12T15:36:19.457Z] Pruning obsolete local branches
[2021-01-12T15:36:19.821Z] Merging remotes/origin/master commit 56bae31eb563fbc532729c13a24f01523e382958 into PR head commit 717c34cb47916406830c7e47889fedc6132ab113
[2021-01-12T15:36:20.297Z] Merge succeeded, producing b6eb776db834f6a7015e7ddabd22fd1d64937289
[2021-01-12T15:36:20.297Z] Checking out Revision b6eb776db834f6a7015e7ddabd22fd1d64937289 (PR-1453)
[2021-01-12T15:36:19.422Z]  > git config remote.origin.url https://github.com/elastic/apm-agent-java.git # timeout=10
[2021-01-12T15:36:19.438Z]  > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
[2021-01-12T15:36:19.448Z]  > git config remote.origin.url https://github.com/elastic/apm-agent-java.git # timeout=10
[2021-01-12T15:36:19.461Z] Fetching upstream changes from https://github.com/elastic/apm-agent-java.git
[2021-01-12T15:36:19.461Z] using GIT_ASKPASS to set credentials GitHub user @apmmachine User + Personal Access Token
[2021-01-12T15:36:19.462Z]  > git fetch --no-tags --progress --prune -- https://github.com/elastic/apm-agent-java.git +refs/pull/1453/head:refs/remotes/origin/PR-1453 +refs/heads/master:refs/remotes/origin/master # timeout=15
[2021-01-12T15:36:19.828Z]  > git config core.sparsecheckout # timeout=10
[2021-01-12T15:36:19.836Z]  > git checkout -f 717c34cb47916406830c7e47889fedc6132ab113 # timeout=15
[2021-01-12T15:36:20.051Z]  > git remote # timeout=10
[2021-01-12T15:36:20.062Z]  > git config --get remote.origin.url # timeout=10
[2021-01-12T15:36:20.071Z] using GIT_ASKPASS to set credentials GitHub user @apmmachine User + Personal Access Token
[2021-01-12T15:36:20.072Z]  > git merge 56bae31eb563fbc532729c13a24f01523e382958 # timeout=10
[2021-01-12T15:36:20.289Z]  > git rev-parse HEAD^{commit} # timeout=10
[2021-01-12T15:36:20.299Z]  > git config core.sparsecheckout # timeout=10
[2021-01-12T15:36:20.319Z]  > git checkout -f b6eb776db834f6a7015e7ddabd22fd1d64937289 # timeout=15
[2021-01-12T15:36:24.097Z] Commit message: "Merge commit '56bae31eb563fbc532729c13a24f01523e382958' into HEAD"
[2021-01-12T15:36:24.115Z] First time build. Skipping changelog.
[2021-01-12T15:36:24.100Z]  > git rev-list --no-walk 6a1b4a639d8073cb1f3fc594d820c1e6cbf0c6c8 # timeout=10
[2021-01-12T15:36:25.048Z] Masking supported pattern matches of $JOB_GCS_BUCKET or $NOTIFY_TO
[2021-01-12T15:36:25.094Z] Timeout set to expire in 1 hr 0 min
[2021-01-12T15:36:25.111Z] The timestamps step is unnecessary when timestamps are enabled for all Pipeline builds.
[2021-01-12T15:36:25.369Z] [INFO] Number of builds to be searched 10
[2021-01-12T15:36:25.644Z] [INFO] 'shallow' is forced to be disabled when running on PullRequests
[2021-01-12T15:36:25.661Z] Running in /var/lib/jenkins/workspace/_java_apm-agent-java-mbp_PR-1453/src/github.com/elastic/apm-agent-java
[2021-01-12T15:36:25.680Z] [INFO] gitCheckout: Checkout SCM PR-1453 with some customisation.
[2021-01-12T15:36:25.708Z] [INFO] Override default checkout
[2021-01-12T15:36:25.742Z] Sleeping for 10 sec
[2021-01-12T15:36:35.875Z] using credential 2a9602aa-ab9f-4e52-baf3-b71ca88469c7-UserAndToken
[2021-01-12T15:36:36.060Z] Cloning the remote Git repository
[2021-01-12T15:36:36.072Z] Cloning repository https://github.com/elastic/apm-agent-java.git
[2021-01-12T15:36:36.097Z]  > git init /var/lib/jenkins/workspace/_java_apm-agent-java-mbp_PR-1453/src/github.com/elastic/apm-agent-java # timeout=10
[2021-01-12T15:36:36.107Z] Using reference repository: /var/lib/jenkins/.git-references/apm-agent-java.git
[2021-01-12T15:36:36.107Z] Fetching upstream changes from https://github.com/elastic/apm-agent-java.git
[2021-01-12T15:36:36.108Z]  > git --version # timeout=10
[2021-01-12T15:36:36.113Z]  > git --version # 'git version 2.17.1'
[2021-01-12T15:36:36.114Z] using GIT_ASKPASS to set credentials GitHub user @apmmachine User + Personal Access Token
[2021-01-12T15:36:36.115Z]  > git fetch --tags --progress -- https://github.com/elastic/apm-agent-java.git +refs/heads/*:refs/remotes/origin/* # timeout=10
[2021-01-12T15:36:36.637Z]  > git config remote.origin.url https://github.com/elastic/apm-agent-java.git # timeout=10
[2021-01-12T15:36:36.643Z]  > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
[2021-01-12T15:36:36.650Z]  > git config remote.origin.url https://github.com/elastic/apm-agent-java.git # timeout=10
[2021-01-12T15:36:36.658Z] Fetching upstream changes from https://github.com/elastic/apm-agent-java.git
[2021-01-12T15:36:36.658Z] using GIT_ASKPASS to set credentials GitHub user @apmmachine User + Personal Access Token
[2021-01-12T15:36:36.659Z]  > git fetch --tags --progress -- https://github.com/elastic/apm-agent-java.git +refs/pull/1453/head:refs/remotes/origin/PR-1453 +refs/heads/master:refs/remotes/origin/master # timeout=10
[2021-01-12T15:36:36.989Z] Checking out Revision 717c34cb47916406830c7e47889fedc6132ab113 (origin/PR-1453)
[2021-01-12T15:36:37.194Z] Commit message: "deleted todo"
[2021-01-12T15:36:37.194Z] First time build. Skipping changelog.
[2021-01-12T15:36:36.982Z]  > git rev-parse origin/PR-1453^{commit} # timeout=10
[2021-01-12T15:36:36.990Z]  > git config core.sparsecheckout # timeout=10
[2021-01-12T15:36:37.003Z]  > git checkout -f 717c34cb47916406830c7e47889fedc6132ab113 # timeout=10
[2021-01-12T15:36:38.395Z] Masking supported pattern matches of $GIT_USERNAME or $GIT_PASSWORD
[2021-01-12T15:36:38.988Z] + git fetch https://****:****@github.com/elastic/apm-agent-java.git +refs/pull/*/head:refs/remotes/origin/pr/*
[2021-01-12T15:36:39.978Z] Running in /var/lib/jenkins/workspace/_java_apm-agent-java-mbp_PR-1453/src/github.com/elastic/apm-agent-java/.git
[2021-01-12T15:36:39.999Z] Archiving artifacts
[2021-01-12T15:36:40.646Z] + git rev-parse HEAD
[2021-01-12T15:36:40.999Z] + git rev-parse HEAD
[2021-01-12T15:36:41.295Z] + git rev-parse origin/pr/1453
[2021-01-12T15:36:41.334Z] [INFO] githubEnv: Found Git Build Cause: pr
[2021-01-12T15:36:41.702Z] Masking supported pattern matches of $GITHUB_TOKEN
[2021-01-12T15:36:42.700Z] [WARN] githubApiCall: The REST API call https://api.github.com/repos/elastic/apm-agent-java/pulls/1453/reviews return 0 elements
[2021-01-12T15:36:42.738Z] [INFO] githubPrCheckApproved: Title: fetch cloud metada - common logic - User: kananindzya - Author Association: CONTRIBUTOR
[2021-01-12T15:36:42.979Z] ERROR: githubPrCheckApproved: The PR is not allowed to run in the CI yet
[2021-01-12T15:36:42.979Z] ERROR: githubPrCheckApproved: The PR is not allowed to run in the CI yet. (Only users with write permissions can do so.)
[2021-01-12T15:36:43.015Z] [INFO] Let's stop build #27. The PR is not allowed to run in the CI yet
[2021-01-12T15:36:43.029Z] Sleeping for 5 sec
[2021-01-12T15:36:44.069Z] Stage "Build" skipped due to earlier failure(s)
[2021-01-12T15:36:44.151Z] Stage "Tests" skipped due to earlier failure(s)
[2021-01-12T15:36:44.197Z] Stage "Unit Tests" skipped due to earlier failure(s)
[2021-01-12T15:36:44.198Z] Stage "Smoke Tests 01" skipped due to earlier failure(s)
[2021-01-12T15:36:44.199Z] Stage "Smoke Tests 02" skipped due to earlier failure(s)
[2021-01-12T15:36:44.199Z] Stage "Benchmarks" skipped due to earlier failure(s)
[2021-01-12T15:36:44.200Z] Stage "Javadoc" skipped due to earlier failure(s)
[2021-01-12T15:36:44.263Z] Failed in branch Unit Tests
[2021-01-12T15:36:44.264Z] Failed in branch Smoke Tests 01
[2021-01-12T15:36:44.264Z] Failed in branch Smoke Tests 02
[2021-01-12T15:36:44.265Z] Failed in branch Benchmarks
[2021-01-12T15:36:44.266Z] Failed in branch Javadoc
[2021-01-12T15:36:44.321Z] Stage "Integration Tests" skipped due to earlier failure(s)
[2021-01-12T15:36:44.358Z] Stage "AfterRelease" skipped due to earlier failure(s)
[2021-01-12T15:36:44.373Z] Stage "AfterRelease" skipped due to earlier failure(s)
[2021-01-12T15:36:44.674Z] Running on Jenkins in /var/lib/jenkins/workspace/_java_apm-agent-java-mbp_PR-1453
[2021-01-12T15:36:44.800Z] [INFO] getVaultSecret: Getting secrets
[2021-01-12T15:36:45.240Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-01-12T15:36:45.896Z] + chmod 755 generate-build-data.sh
[2021-01-12T15:36:45.896Z] + ./generate-build-data.sh https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-java/apm-agent-java-mbp/PR-1453/ https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-java/apm-agent-java-mbp/PR-1453/runs/27 ABORTED 169589
[2021-01-12T15:36:46.147Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-java/apm-agent-java-mbp/PR-1453/runs/27/steps/?limit=10000 -o steps-info.json
[2021-01-12T15:36:47.058Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-java/apm-agent-java-mbp/PR-1453/runs/27/tests/?status=FAILED -o tests-errors.json
[2021-01-12T15:36:47.058Z] Retry 1/3 exited 22, retrying in 1 seconds...
[2021-01-12T15:36:48.401Z] Retry 2/3 exited 22, retrying in 2 seconds...
[2021-01-12T15:36:50.262Z] Retry 3/3 exited 22, no more retries left.
[2021-01-12T15:36:50.262Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-java/apm-agent-java-mbp/PR-1453/runs/27/log/ -o pipeline-log.txt

videnkz · 2020-11-11T17:28:18Z

this work only with apm-server 7.10.0 version:

videnkz · 2020-11-12T05:28:49Z

Hi @eyalkoren , please review this pull request.
Thanks

codecov-io · 2020-11-16T08:42:49Z

Codecov Report

Merging #1453 (717c34c) into master (fc1a0f4) will decrease coverage by 0.09%.
The diff coverage is 70.74%.

@@             Coverage Diff              @@
##             master    #1453      +/-   ##
============================================
- Coverage     71.41%   71.32%   -0.10%     
- Complexity     2583     2614      +31     
============================================
  Files           252      254       +2     
  Lines         11221    11489     +268     
  Branches       1460     1502      +42     
============================================
+ Hits           8014     8194     +180     
- Misses         2688     2737      +49     
- Partials        519      558      +39

Impacted Files	Coverage Δ	Complexity Δ
...src/main/java/co/elastic/apm/agent/impl/Cloud.java	`61.29% <61.29%> (ø)`	`11.00 <11.00> (?)`
...stic/apm/agent/impl/payload/CloudProviderInfo.java	`69.38% <69.38%> (ø)`	`16.00 <16.00> (?)`
.../apm/agent/report/serialize/DslJsonSerializer.java	`89.51% <97.22%> (+0.33%)`	`193.00 <6.00> (+7.00)`
...tic/apm/agent/configuration/CoreConfiguration.java	`96.88% <100.00%> (+0.07%)`	`39.00 <1.00> (+1.00)`
.../main/java/co/elastic/apm/agent/impl/MetaData.java	`95.45% <100.00%> (+0.45%)`	`9.00 <2.00> (+1.00)`
...iguration/PrefixingConfigurationSourceWrapper.java	`50.00% <0.00%> (-16.67%)`	`3.00% <0.00%> (-1.00%)`
...agent/report/serialize/MetricRegistryReporter.java	`86.66% <0.00%> (-6.67%)`	`4.00% <0.00%> (-1.00%)`
...ration/source/PropertyFileConfigurationSource.java	`71.42% <0.00%> (-5.72%)`	`10.00% <0.00%> (-1.00%)`
...lastic/apm/agent/impl/ElasticApmTracerBuilder.java	`73.33% <0.00%> (-4.45%)`	`20.00% <0.00%> (ø%)`
...ent/report/serialize/MetricRegistrySerializer.java	`87.60% <0.00%> (-2.48%)`	`30.00% <0.00%> (-2.00%)`
... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fc1a0f4...717c34c. Read the comment docs.

eyalkoren · 2020-11-24T07:05:13Z

@kananindzya don't worry, I didn't forget about it, it just takes some time to get to it...
I will probably do a PR against your PR, suggesting some changes

videnkz marked this pull request as draft October 24, 2020 21:16

eyalkoren added the [zube]: In Progress label Oct 26, 2020

videnkz marked this pull request as ready for review November 11, 2020 17:39

videnkz changed the title ~~WIP: fetch cloud metada - common logic~~ fetch cloud metada - common logic Nov 11, 2020

videnkz added 13 commits November 13, 2020 01:33

fetch cloud metada - common logic

9b6a869

implement aws metadata fetch

4ddc982

added gcp metadata parse

712c25b

added azure metadata parse

daba9be

added cloud metadata serialization

46fb991

minor fixes, added test for gcp deserialize

313fbe3

minor fixes

5a348e6

added comma in serialization

6ea891f

fixes with serialization, added tests

02c49d2

deleted mock aws url

6ece6f4

added name to account

a8aad17

added ascii docs

a657c21

deleted todo

717c34c

videnkz force-pushed the issue-1264-collect-cloud-metadata branch from 940b15e to 717c34c Compare November 12, 2020 19:33

eyalkoren added the agent-java label Nov 17, 2020

eyalkoren self-assigned this Nov 24, 2020

eyalkoren mentioned this pull request Dec 20, 2020

[META 388] Collect Azure App Service cloud metadata #1578

Open

eyalkoren mentioned this pull request Jan 6, 2021

Cloud metadata #1599

Merged

10 tasks

videnkz closed this Jan 18, 2021

zube bot added [zube]: Done and removed [zube]: In Progress labels Jan 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fetch cloud metada - common logic #1453

fetch cloud metada - common logic #1453

videnkz commented Oct 24, 2020 •

edited

Loading

apmmachine commented Oct 24, 2020 •

edited

Loading

Build stats

`Error signal`

videnkz commented Nov 11, 2020

videnkz commented Nov 12, 2020

codecov-io commented Nov 16, 2020

eyalkoren commented Nov 24, 2020

fetch cloud metada - common logic #1453

fetch cloud metada - common logic #1453

Conversation

videnkz commented Oct 24, 2020 • edited Loading

apmmachine commented Oct 24, 2020 • edited Loading

❕ Build Aborted

Build stats

Steps errors

Error signal

Log output

videnkz commented Nov 11, 2020

videnkz commented Nov 12, 2020

codecov-io commented Nov 16, 2020

Codecov Report

eyalkoren commented Nov 24, 2020

videnkz commented Oct 24, 2020 •

edited

Loading

apmmachine commented Oct 24, 2020 •

edited

Loading

`Error signal`