Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Filebeat] Fix hardcoded amazonaws.com endpoint #24861

Merged
merged 4 commits into from
Apr 9, 2021
Merged

[Filebeat] Fix hardcoded amazonaws.com endpoint #24861

merged 4 commits into from
Apr 9, 2021

Conversation

legoguy1000
Copy link
Contributor

@legoguy1000 legoguy1000 commented Mar 30, 2021 •
edited
Loading

What does this PR do?

Updates the S3 input validation for the queue url to check for the amazonaws.com domain or for the domain to match the endpoint config parameter that already exists.

Why is it important?

Adds the ability for to use non amazonaws.com domains for Filebeat S3 input.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

How to test this PR locally

Related issues

closes #24420

Use cases

Screenshots

Logs

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Mar 30, 2021
@legoguy1000 legoguy1000 marked this pull request as ready for review March 30, 2021 23:57
@legoguy1000
Copy link
Contributor Author

Besides running the standard pythonIntegTest I'm not sure how to best test changes to the Go code besides trying to spin up the binary and test the configs.

@elasticmachine
Copy link
Collaborator

elasticmachine commented Mar 31, 2021 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: kaiyan-sheng commented: /test

  • Start Time: 2021-04-08T16:11:55.980+0000

  • Duration: 50 min 2 sec

  • Commit: 85d5b0f

Test stats 🧪

Test Results
Failed 0
Passed 6910
Skipped 1185
Total 8095

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 6910
Skipped 1185
Total 8095

@kaiyan-sheng kaiyan-sheng self-assigned this Mar 31, 2021
@kaiyan-sheng kaiyan-sheng added the Team:Integrations Label for the Integrations team label Mar 31, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations (Team:Integrations)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Mar 31, 2021
@kaiyan-sheng kaiyan-sheng added needs_backport PR is waiting to be backported to other branches. needs_team Indicates that the issue/PR needs a Team:* label labels Mar 31, 2021
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Mar 31, 2021
@botelastic
Copy link

botelastic bot commented Mar 31, 2021

This pull request doesn't have a Team:<team> label.

@kaiyan-sheng kaiyan-sheng added the test-plan Add this PR to be manual test plan label Mar 31, 2021
@kaiyan-sheng
Copy link
Contributor

Besides running the standard pythonIntegTest I'm not sure how to best test changes to the Go code besides trying to spin up the binary and test the configs.

@legoguy1000 Thanks for your contribution! We have unit tests for the golang code and for this function getRegionFromQueueURL specifically, there is TestGetRegionFromQueueURL under beats/x-pack/filebeat/input/awss3/collector_test.go. Please add some test cases there for URLs that have a different endpoint. Thank you!!

@legoguy1000
Copy link
Contributor Author

Besides running the standard pythonIntegTest I'm not sure how to best test changes to the Go code besides trying to spin up the binary and test the configs.

@legoguy1000 Thanks for your contribution! We have unit tests for the golang code and for this function getRegionFromQueueURL specifically, there is TestGetRegionFromQueueURL under beats/x-pack/filebeat/input/awss3/collector_test.go. Please add some test cases there for URLs that have a different endpoint. Thank you!!

Done.

@legoguy1000 legoguy1000 requested a review from kaiyan-sheng April 2, 2021 11:50
@legoguy1000 legoguy1000 requested a review from kaiyan-sheng April 7, 2021 13:29
@kaiyan-sheng
Copy link
Contributor

/test

kaiyan-sheng
kaiyan-sheng approved these changes Apr 9, 2021
View reviewed changes
Copy link
Contributor

@kaiyan-sheng kaiyan-sheng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your contribution!! This PR looks good to me!

@kaiyan-sheng kaiyan-sheng merged commit 7b729da into elastic:master Apr 9, 2021
@kaiyan-sheng kaiyan-sheng removed the test-plan Add this PR to be manual test plan label Apr 9, 2021
@kaiyan-sheng kaiyan-sheng mentioned this pull request Apr 9, 2021
Merged
6 tasks
@kaiyan-sheng kaiyan-sheng added v7.13.0 and removed needs_backport PR is waiting to be backported to other branches. labels Apr 9, 2021
@kaiyan-sheng kaiyan-sheng mentioned this pull request Apr 9, 2021
Merged
6 tasks
kaiyan-sheng added a commit that referenced this pull request Apr 10, 2021
@kaiyan-sheng @legoguy1000
[Filebeat] Fix hardcoded amazonaws.com endpoint (#24861) (#25009)
7428953 
(cherry picked from commit 7b729da)

Co-authored-by: Alex Resnick <[email protected]>
kaiyan-sheng added a commit that referenced this pull request Apr 10, 2021
@kaiyan-sheng @legoguy1000
[Filebeat] Fix hardcoded amazonaws.com endpoint (#24861) (#25010)
34a84af 
(cherry picked from commit 7b729da)

Co-authored-by: Alex Resnick <[email protected]>
v1v added a commit to v1v/beats that referenced this pull request Apr 14, 2021
@v1v
Merge remote-tracking branch 'upstream/master' into feature/use-debian-9
1de2782 
* upstream/master: (308 commits)
  [winlogbeat] Add support for sysmon v13 events 24 and 25 (elastic#24945)
  mergify: add backport label (elastic#25050)
  Add pod.ip in k8s metadata (elastic#25037)
  [elastic-agent] Use fleet.url for container cmd (elastic#25026)
  disable TestXPackEnabled flaky test in logstash metricbeat module (elastic#25034)
  Leverege leader election in agent  k8s manifests (elastic#25016)
  libbeat/publisher/pipeline: expand monitoring (elastic#24700)
  libbeat: fix decode_json_fields config validation (elastic#24862)
  Remove make docs-preview instructions (elastic#25001)
  [Filebeat] Fix IPtables pipeline (elastic#24928)
  [DOCS] cd into correct directory before invoking mage. (elastic#17679)
  Add -buildmode=pie for supported platform (elastic#24964)
  Add agent's direcotry in k8s manifest generator (elastic#24987)
  [mergify] assign the original author (elastic#25007)
  Fix AWS module flaky tests (elastic#24852)
  [filebeat] Use fail_on_template_error on google_workspace and okta pagination (elastic#24967)
  Updated config to match defaults (elastic#25004)
  [Filebeat] Fix hardcoded amazonaws.com endpoint (elastic#24861)
  Add cloud.service.name to add_cloud_metadata (elastic#24993)
  [Ingest Manager] Expose processes and their metrics (elastic#24788)
  ...
@legoguy1000 legoguy1000 deleted the 24420-s3-endpoint branch April 29, 2021 00:39
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
@kaiyan-sheng @legoguy1000
[Filebeat] Fix hardcoded amazonaws.com endpoint (elastic#24861) (elas…
cb71daf 
…tic#25010)

(cherry picked from commit 7129c21)

Co-authored-by: Alex Resnick <[email protected]>
@strawgate strawgate mentioned this pull request May 24, 2024
Closed
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kaiyan-sheng kaiyan-sheng kaiyan-sheng approved these changes

Assignees

@kaiyan-sheng kaiyan-sheng

Labels
Team:Integrations Label for the Integrations team v7.12.1 v7.13.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Filebeat AWS Module S3 input error queueURL is not in format
3 participants
@legoguy1000 @elasticmachine @kaiyan-sheng