Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cherry-pick #19423 to 7.x: [MetricBeat] add param aws_partition to support aws-cn, aws-us-gov regions #19519

Merged
merged 4 commits into from
Jul 1, 2020
Merged

Cherry-pick #19423 to 7.x: [MetricBeat] add param aws_partition to support aws-cn, aws-us-gov regions #19519

merged 4 commits into from
Jul 1, 2020

Conversation

kaiyan-sheng
Copy link
Contributor

@kaiyan-sheng kaiyan-sheng commented Jun 30, 2020 •
edited by zube bot
Loading

Cherry-pick of PR #19423 to 7.x branch. Original message:

What does this PR do?

This PR is to add a parameter aws_partition for metricbeat aws cloudwatch module, so metricbeat can support aws china and aws us-gov account.

Why is it important?

Because there are three partitions aws, aws-cn, aws-us-gov of aws regions, now metricbeat can only support aws global region, here is a hardcode.

beats/x-pack/metricbeat/module/aws/aws.go

Lines 95 to 100 in 5ba7026

// Get IAM account name
awsConfig.Region = "us-east-1"
svcIam := iam.New(awscommon.EnrichAWSConfigWithEndpoint(
config.AWSConfig.Endpoint, "iam", "", awsConfig))
req := svcIam.ListAccountAliasesRequest(&iam.ListAccountAliasesInput{})

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

  1. add aws_partition param in aws module config
metricbeat.modules:
  - module: aws
    aws_partition: aws-cn
    period: 300s
    access_key_id: 
    secret_access_key:
  1. run metricbeat , open debug log, then you can see it can retrive metrics from correct regions

Related issues

Use cases

  • no side affect
  • user can use metricbeat to get metrics from aws china and aws us-gov account except aws global account

Screenshots

aws global account, set aws_partition: aws in aws module config or remove the param

image

aws china account, set aws_partition: aws-cn in aws module config

image

@kwinstonix @kaiyan-sheng
[MetricBeat] add param aws_partition to support aws-cn, aws-us-gov …
9fd31ae 
…regions (#19423)

* add params `aws_partition` to support other aws region: aws-cn, aws-us-gov

(cherry picked from commit 1ea8c84)
@kaiyan-sheng kaiyan-sheng added [zube]: In Review backport Team:Platforms Label for the Integrations - Platforms team labels Jun 30, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-platforms (Team:Platforms)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 30, 2020
@kaiyan-sheng kaiyan-sheng self-assigned this Jun 30, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 30, 2020 •
edited
Loading

❕ Build Aborted

There is a new build on-going so the previous on-going builds have been aborted.

Pipeline View Test View Changes Artifacts

Expand to view the summary

Build stats

  • Build Cause: [Pull request #19519 updated]

  • Reason: Aborted from #6

  • Start Time: 2020-07-01T13:47:20.258+0000

  • Duration: 111 min 26 sec

  • Commit: 1d53741

Test stats 🧪

Test Results
Failed 0
Passed 3680
Skipped 512
Total 4192

Log output

Expand to view the last 100 lines of log output 

[2020-07-01T15:04:22.791Z] WARNING: You are using pip version 19.2.3, however version 20.1.1 is available.
[2020-07-01T15:04:22.791Z] You should consider upgrading via the 'python -m pip install --upgrade pip' command.
[2020-07-01T15:04:22.791Z] S..
[2020-07-01T15:04:22.791Z] [success] 74.30% test_base.Test.test_export_function: 0.3363s
[2020-07-01T15:04:22.791Z] [success] 25.70% test_base.Test.test_export_function_invalid_conf: 0.1163s
[2020-07-01T15:04:22.791Z] ----------------------------------------------------------------------
[2020-07-01T15:04:22.791Z] Ran 3 tests in 0.454s
[2020-07-01T15:04:22.791Z] 
[2020-07-01T15:04:22.791Z] OK (SKIP=1)
[2020-07-01T15:04:22.791Z] >> python test: Unit Testing Complete
[2020-07-01T15:04:22.861Z] Recording test results
[2020-07-01T15:04:25.241Z] Stashed 2 file(s)
[2020-07-01T15:04:25.251Z] Archiving artifacts
[2020-07-01T15:38:02.174Z] .......Sending interrupt signal to process
[2020-07-01T15:38:04.906Z] Sending interrupt signal to process
[2020-07-01T15:38:17.170Z] Click here to forcibly terminate running steps
[2020-07-01T15:38:19.823Z] failed to run compiled magefile: signal: terminated
[2020-07-01T15:38:19.831Z] script returned exit code 255
[2020-07-01T15:38:19.912Z] Recording test results
[2020-07-01T15:38:21.596Z] Stashed 3 file(s)
[2020-07-01T15:38:21.605Z] Archiving artifacts
[2020-07-01T15:38:22.409Z] + curl -sSLo codecov https://codecov.io/bash
[2020-07-01T15:38:22.669Z] + FILE=auditbeat/build/coverage/full.cov
[2020-07-01T15:38:22.669Z] + [ -f auditbeat/build/coverage/full.cov ]
[2020-07-01T15:38:22.669Z] + FILE=filebeat/build/coverage/full.cov
[2020-07-01T15:38:22.669Z] + [ -f filebeat/build/coverage/full.cov ]
[2020-07-01T15:38:22.669Z] + FILE=heartbeat/build/coverage/full.cov
[2020-07-01T15:38:22.669Z] + [ -f heartbeat/build/coverage/full.cov ]
[2020-07-01T15:38:22.669Z] + FILE=libbeat/build/coverage/full.cov
[2020-07-01T15:38:22.669Z] + [ -f libbeat/build/coverage/full.cov ]
[2020-07-01T15:38:22.669Z] + FILE=metricbeat/build/coverage/full.cov
[2020-07-01T15:38:22.669Z] + [ -f metricbeat/build/coverage/full.cov ]
[2020-07-01T15:38:22.669Z] + FILE=packetbeat/build/coverage/full.cov
[2020-07-01T15:38:22.669Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-07-01T15:38:22.669Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-07-01T15:38:22.669Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-07-01T15:38:22.669Z] + FILE=journalbeat/build/coverage/full.cov
[2020-07-01T15:38:22.669Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-07-01T15:38:24.052Z] Failed in branch Filebeat x-pack
[2020-07-01T15:38:24.207Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats
[2020-07-01T15:38:24.522Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-07-01T15:38:24.533Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Lint
[2020-07-01T15:38:24.612Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-07-01T15:38:24.686Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Dockerlogbeat
[2020-07-01T15:38:24.760Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-07-01T15:38:24.845Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Elastic-Agent-x-pack-Windows
[2020-07-01T15:38:24.928Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-07-01T15:38:25.002Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Auditbeat-x-pack-Windows
[2020-07-01T15:38:25.080Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-07-01T15:38:25.167Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-07-01T15:38:25.244Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Libbeat-x-pack
[2020-07-01T15:38:25.319Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-07-01T15:38:25.394Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Elastic-Agent-Mac-OS-X
[2020-07-01T15:38:25.471Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-07-01T15:38:25.548Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Auditbeat-x-pack-Mac-OS-X
[2020-07-01T15:38:25.633Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-07-01T15:38:25.710Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-07-01T15:38:25.800Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Functionbeat-Mac-OS-X-x-pack
[2020-07-01T15:38:25.873Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Functionbeat-Windows
[2020-07-01T15:38:25.957Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Filebeat-x-pack
[2020-07-01T15:38:26.324Z] + cat
[2020-07-01T15:38:26.324Z] + /usr/local/bin/runbld ./runbld-script
[2020-07-01T15:38:26.324Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-07-01T15:38:32.926Z] runbld>>> runbld started
[2020-07-01T15:38:32.926Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-07-01T15:38:33.868Z] runbld>>> The following profiles matched the job 'Beats/beats-beats-mbp/PR-19519' in order of occurrence in the config (last value wins).
[2020-07-01T15:38:35.257Z] runbld>>> Debug logging enabled.
[2020-07-01T15:38:35.257Z] runbld>>> Storing result
[2020-07-01T15:38:35.257Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-07-01T15:38:35.257Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200701153834-479C418D
[2020-07-01T15:38:35.257Z] runbld>>> Adding system facts.
[2020-07-01T15:38:36.201Z] runbld>>> Adding vcs info for the latest commit:  bc31a955805ee892da6a6ebd2b5ee3a322506299
[2020-07-01T15:38:36.201Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-07-01T15:38:36.201Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-07-01T15:38:36.201Z] Processing JUnit reports with runbld...
[2020-07-01T15:38:36.201Z] + echo 'Processing JUnit reports with runbld...'
[2020-07-01T15:38:36.461Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-07-01T15:38:36.461Z] runbld>>> DURATION: 18ms
[2020-07-01T15:38:36.461Z] runbld>>> STDOUT: 40 bytes
[2020-07-01T15:38:36.461Z] runbld>>> STDERR: 49 bytes
[2020-07-01T15:38:36.461Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-07-01T15:38:36.461Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats
[2020-07-01T15:38:37.403Z] runbld>>> Storing build metadata: 
[2020-07-01T15:38:37.403Z] runbld>>> Adding test report.
[2020-07-01T15:38:37.403Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats
[2020-07-01T15:38:38.343Z] runbld>>> Found 30 test output files
[2020-07-01T15:38:39.283Z] runbld>>> Test output logs contained: Errors: 0 Failures: 0 Tests: 4042 Skipped: 475
[2020-07-01T15:38:39.283Z] runbld>>> Storing result
[2020-07-01T15:38:39.283Z] runbld>>> FAILURES: 0
[2020-07-01T15:38:39.543Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-07-01T15:38:39.543Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200701153834-479C418D
[2020-07-01T15:38:39.543Z] runbld>>> Email notification disabled by environment variable.
[2020-07-01T15:38:39.543Z] runbld>>> Slack notification disabled by environment variable.
[2020-07-01T15:38:45.157Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519
[2020-07-01T15:38:45.302Z] [INFO] getVaultSecret: Getting secrets
[2020-07-01T15:38:45.388Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-07-01T15:38:46.500Z] + chmod 755 generate-build-data.sh
[2020-07-01T15:38:46.500Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19519/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19519/runs/5 ABORTED 6685973
[2020-07-01T15:38:46.500Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19519/runs/5/steps/?limit=10000 -o steps-info.json
[2020-07-01T15:38:47.843Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19519/runs/5/tests/?status=FAILED -o tests-errors.json

@kaiyan-sheng
Copy link
Contributor Author

CI failure should be fixed by #19521

@kaiyan-sheng
Copy link
Contributor Author

run tests please

ycombinator
ycombinator approved these changes Jul 1, 2020
View reviewed changes
Copy link
Contributor

@ycombinator ycombinator left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@kaiyan-sheng kaiyan-sheng merged commit 60ea971 into elastic:7.x Jul 1, 2020
@kaiyan-sheng kaiyan-sheng deleted the backport_19423_7.x branch July 1, 2020 16:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ycombinator ycombinator ycombinator approved these changes

Assignees

@kaiyan-sheng kaiyan-sheng

Labels
backport Team:Platforms Label for the Integrations - Platforms team [zube]: Done
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@kaiyan-sheng @elasticmachine @ycombinator @kwinstonix