Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cherry-pick #15668 to 7.7: Support application default credentials (ADC) for Google Pub/Sub #17791

Merged
merged 3 commits into from
May 13, 2020

Conversation

andrewkroh
Copy link
Member

Cherry-pick of PR #15668 to 7.7 branch. Original message:

What does this PR do?

Update the Google Pub/Sub input to support reading Application Default Credentials
(ADC) in addition to the credentials_file and credentials_json config options.

If neither config option is set then it will attempt to search for the default credentials.
Generally this means reading the GOOGLE_APPLICATION_CREDENTIALS environment
variable plus searching a few other well known locations.

The googlecloud module was updates to support all three authentication mechanisms.

Why is it important?

Without this change, Filebeat requires a path to a file with credentials to be provided in its config and is unable to use for example a default service account.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works

Author's Checklist

  • [ ]

How to test this PR locally

  1. Upload the Filebeat binary to a GCP instance created with default config
  2. Run the binary configured to read from PubSub, but do not provide any credentials
  3. Filebeat should be able to detect that it's on GCP and use ADC (get a token for the default service account for the VM from the metadata server)

Related issues

Use cases

Sending logs to Elastic from PubSub queue

Screenshots

Logs

@elasticmachine
Copy link
Collaborator

Pinging @elastic/siem (Team:SIEM)

@andrewkroh andrewkroh force-pushed the backport_15668_7.7 branch from fc04604 to fcc6ac4 Compare May 2, 2020 02:41
…stic#15668)

Update the Google Pub/Sub input to support reading Application Default Credentials
(ADC) in addition to the credentials_file and credentials_json config options.

If neither config option is set then it will attempt to search for the default credentials.
Generally this means reading the GOOGLE_APPLICATION_CREDENTIALS environment
variable plus search a few other well known locations.

The googlecloud module was updates to support all three authentication mechanisms.

Co-authored-by: Michal Wasilewski <[email protected]>
Co-authored-by: Andrew Kroh <[email protected]>
(cherry picked from commit 058de35)
@andrewkroh andrewkroh force-pushed the backport_15668_7.7 branch from fcc6ac4 to b087e35 Compare May 2, 2020 02:46
@andrewkroh andrewkroh requested a review from kaiyan-sheng May 2, 2020 02:47
Copy link
Contributor

@leehinman leehinman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@andrewkroh
Copy link
Member Author

run tests

@andrewkroh andrewkroh merged commit 26cc4ad into elastic:7.7 May 13, 2020
@andrewkroh andrewkroh deleted the backport_15668_7.7 branch January 14, 2022 14:04
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
…stic#15668) (elastic#17791)

Update the Google Pub/Sub input to support reading Application Default Credentials
(ADC) in addition to the credentials_file and credentials_json config options.

If neither config option is set then it will attempt to search for the default credentials.
Generally this means reading the GOOGLE_APPLICATION_CREDENTIALS environment
variable plus search a few other well known locations.

The googlecloud module was updates to support all three authentication mechanisms.

Co-authored-by: Michal Wasilewski <[email protected]>
Co-authored-by: Andrew Kroh <[email protected]>
(cherry picked from commit e3175aa)

Co-authored-by: Michal <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants