Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add additional fields to address issue #18465 for googlecloud audit log #18472

Merged
merged 4 commits into from
May 14, 2020
Merged

Add additional fields to address issue #18465 for googlecloud audit log #18472

merged 4 commits into from
May 14, 2020

Conversation

alakahakai
Copy link

@alakahakai alakahakai commented May 13, 2020

Address issue #18465

The returned fields from googlecloud audit logs have changed recently. These fields don't have defined data models and just use a struct, which caused the issue. With this PR, the googlecloud.audit dataset now only take in the fields that are defined by the dataset.

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label May 13, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/siem (Team:SIEM)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label May 13, 2020
@alakahakai alakahakai requested a review from andrewkroh May 13, 2020 01:59
@alakahakai alakahakai changed the title Address issue 18465 to add additional fields for googlecloud audit log Add additional fields to address issue 18465 for googlecloud audit log May 13, 2020
@alakahakai alakahakai changed the title Add additional fields to address issue 18465 for googlecloud audit log Add additional fields to address issue #18465 for googlecloud audit log May 13, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented May 13, 2020

❕ Build Aborted

There is a new build on-going so the previous on-going builds have been aborted.

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

Log output

Expand to view the last 100 lines of log output

[2020-05-14T22:39:22.467Z] Stage "Auditbeat oss" skipped due to when conditional
[2020-05-14T22:39:22.468Z] Stage "Libbeat" skipped due to when conditional
[2020-05-14T22:39:22.468Z] Stage "Functionbeat" skipped due to when conditional
[2020-05-14T22:39:22.469Z] Stage "Generators" skipped due to when conditional
[2020-05-14T22:39:22.668Z] Stage "Auditbeat oss" skipped due to when conditional
[2020-05-14T22:39:22.670Z] Stage "Generators" skipped due to when conditional
[2020-05-14T22:39:36.685Z] Still waiting to schedule task
[2020-05-14T22:39:36.686Z] All nodes of label ‘ubuntu&&immutable’ are offline
[2020-05-14T22:39:36.688Z] Still waiting to schedule task
[2020-05-14T22:39:36.688Z] All nodes of label ‘ubuntu&&immutable’ are offline
[2020-05-14T22:39:36.692Z] Still waiting to schedule task
[2020-05-14T22:39:36.693Z] ‘beats-ci-immutable-windows-2019-1589494613632667519’ is offline
[2020-05-14T22:40:07.904Z] Running on beats-ci-immutable-ubuntu-1604-1589495965850127510 in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18472
[2020-05-14T22:40:09.313Z] [INFO] unstashV2: JOB_GCS_BUCKET is set. bucket param got precedency instead.
[2020-05-14T22:40:09.338Z] [INFO] unstashV2: JOB_GCS_CREDENTIALS is set. credentialsId param got precedency instead.
[2020-05-14T22:40:09.407Z] [Google Cloud Storage Plugin] Found 1 files to download from pattern: gs://beats-ci-temp/Beats/beats-beats-mbp/PR-18472-4/source/source.tgz
[2020-05-14T22:40:11.129Z] [Google Cloud Storage Plugin] Downloading: Beats/beats-beats-mbp/PR-18472-4/source/source.tgz to local path: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18472/source.tgz
[2020-05-14T22:40:13.069Z] Running on beats-ci-immutable-ubuntu-1604-1589495965850148444 in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18472
[2020-05-14T22:40:14.224Z] [INFO] unstashV2: JOB_GCS_BUCKET is set. bucket param got precedency instead.
[2020-05-14T22:40:14.248Z] [INFO] unstashV2: JOB_GCS_CREDENTIALS is set. credentialsId param got precedency instead.
[2020-05-14T22:40:14.325Z] [Google Cloud Storage Plugin] Found 1 files to download from pattern: gs://beats-ci-temp/Beats/beats-beats-mbp/PR-18472-4/source/source.tgz
[2020-05-14T22:40:16.219Z] [Google Cloud Storage Plugin] Downloading: Beats/beats-beats-mbp/PR-18472-4/source/source.tgz to local path: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18472/source.tgz
[2020-05-14T22:40:22.005Z] + tar -xpf source.tgz
[2020-05-14T22:40:25.273Z] [INFO] unstashV2: JOB_GCS_BUCKET is set. bucket param got precedency instead.
[2020-05-14T22:40:25.295Z] [INFO] unstashV2: JOB_GCS_CREDENTIALS is set. credentialsId param got precedency instead.
[2020-05-14T22:40:25.401Z] [Google Cloud Storage Plugin] Found 1 files to download from pattern: gs://beats-ci-temp/Beats/beats-beats-mbp/PR-18472-4/source/source.tgz
[2020-05-14T22:40:25.491Z] [Google Cloud Storage Plugin] Downloading: Beats/beats-beats-mbp/PR-18472-4/source/source.tgz to local path: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18472/source.tgz
[2020-05-14T22:40:26.851Z] + tar -xpf source.tgz
[2020-05-14T22:40:32.445Z] + rm source.tgz
[2020-05-14T22:40:32.755Z] + command -v docker
[2020-05-14T22:40:32.756Z] /usr/bin/docker
[2020-05-14T22:40:32.770Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18472/src/github.com/elastic/beats
[2020-05-14T22:40:33.128Z] + .ci/scripts/install-go.sh
[2020-05-14T22:40:33.128Z] + MSG='parameter missing.'
[2020-05-14T22:40:33.128Z] + GO_VERSION=1.13.10
[2020-05-14T22:40:33.128Z] + PROPERTIES_FILE=go_env.properties
[2020-05-14T22:40:33.128Z] + HOME=/var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18472
[2020-05-14T22:40:33.128Z] ++ tr '[:upper:]' '[:lower:]'
[2020-05-14T22:40:33.128Z] ++ uname -s
[2020-05-14T22:40:33.128Z] + ARCH=linux
[2020-05-14T22:40:33.128Z] + GVM_CMD=/var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18472/bin/gvm
[2020-05-14T22:40:33.128Z] + mkdir -p /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18472/bin
[2020-05-14T22:40:33.128Z] + curl -sSLo /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18472/bin/gvm https://github.com/andrewkroh/gvm/releases/download/v0.2.2/gvm-linux-amd64
[2020-05-14T22:40:34.676Z] Sending interrupt signal to process
[2020-05-14T22:40:36.761Z] Sending interrupt signal to process
[2020-05-14T22:40:36.879Z] tar: Unexpected EOF in archive
[2020-05-14T22:40:36.879Z] tar: Unexpected EOF in archive
[2020-05-14T22:40:36.879Z] tar: Error is not recoverable: exiting now
[2020-05-14T22:40:36.916Z] script returned exit code 2
[2020-05-14T22:40:38.611Z] Sending interrupt signal to process
[2020-05-14T22:40:38.744Z] Failed in branch Filebeat Windows
[2020-05-14T22:40:39.734Z] .ci/scripts/install-go.sh: line 13:  2423 Terminated              curl -sSLo "${GVM_CMD}" "https://github.com/andrewkroh/gvm/releases/download/v0.2.2/gvm-${ARCH}-amd64"
[2020-05-14T22:40:39.751Z] script returned exit code 143
[2020-05-14T22:40:39.838Z] Failed in branch Filebeat Mac OS X
[2020-05-14T22:40:39.958Z] Failed in branch Filebeat x-pack
[2020-05-14T22:40:41.284Z] Failed in branch Filebeat oss
[2020-05-14T22:40:41.424Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18472/src/github.com/elastic/beats
[2020-05-14T22:40:41.747Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-05-14T22:40:41.761Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18472/src/github.com/elastic/beats/Lint
[2020-05-14T22:40:42.145Z] + cat
[2020-05-14T22:40:42.145Z] + /usr/local/bin/runbld ./runbld-script
[2020-05-14T22:40:42.145Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-05-14T22:40:48.750Z] runbld>>> runbld started
[2020-05-14T22:40:48.750Z] runbld>>> 1.6.11/a66728ff8f4356963772e6e6d2069392fa06acbe
[2020-05-14T22:40:49.672Z] Click here to forcibly terminate running steps
[2020-05-14T22:40:50.677Z] runbld>>> The following profiles matched the job 'Beats/beats-beats-mbp/PR-18472' in order of occurrence in the config (last value wins).
[2020-05-14T22:40:51.622Z] runbld>>> Debug logging enabled.
[2020-05-14T22:40:51.885Z] runbld>>> Storing result
[2020-05-14T22:40:51.885Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-05-14T22:40:51.885Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200514224051-9A3997A8
[2020-05-14T22:40:51.885Z] runbld>>> Adding system facts.
[2020-05-14T22:40:50.064Z] [Google Cloud Storage Plugin] Downloading: Beats/beats-beats-mbp/PR-18472-4/source/source.tgz to local path: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18472/source.tgz
[2020-05-14T22:40:52.831Z] runbld>>> Adding vcs info for the latest commit:  34a1cb71cb8170429f2bff804c22636b19dd2468
[2020-05-14T22:40:53.093Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-05-14T22:40:53.093Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-05-14T22:40:53.093Z] Processing JUnit reports with runbld...
[2020-05-14T22:40:53.093Z] + echo 'Processing JUnit reports with runbld...'
[2020-05-14T22:40:53.355Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-05-14T22:40:53.355Z] runbld>>> DURATION: 13ms
[2020-05-14T22:40:53.355Z] runbld>>> STDOUT: 40 bytes
[2020-05-14T22:40:53.355Z] runbld>>> STDERR: 49 bytes
[2020-05-14T22:40:53.355Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-05-14T22:40:53.355Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18472/src/github.com/elastic/beats
[2020-05-14T22:40:54.755Z] runbld>>> Storing build metadata: 
[2020-05-14T22:40:54.755Z] runbld>>> Adding test report.
[2020-05-14T22:40:54.755Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18472/src/github.com/elastic/beats
[2020-05-14T22:40:56.146Z] runbld>>> Found 0 test output files
[2020-05-14T22:40:56.146Z] runbld>>> Test output logs contained: Errors: 0 Failures: 0 Tests: 0 Skipped: 0
[2020-05-14T22:40:56.146Z] runbld>>> Storing result
[2020-05-14T22:40:56.146Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-05-14T22:40:56.146Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200514224051-9A3997A8
[2020-05-14T22:40:56.146Z] runbld>>> Email notification disabled by environment variable.
[2020-05-14T22:40:56.146Z] runbld>>> Slack notification disabled by environment variable.
[2020-05-14T22:41:02.069Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18472
[2020-05-14T22:41:02.458Z] [INFO] getVaultSecret: Getting secrets
[2020-05-14T22:41:03.058Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-05-14T22:41:04.012Z] + chmod 755 generate-build-data.sh
[2020-05-14T22:41:04.012Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18472/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18472/runs/4 ABORTED 1368129
[2020-05-14T22:41:04.563Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18472/runs/4/steps/?limit=10000 -o steps-info.json
[2020-05-14T22:41:05.115Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18472/runs/4/tests/?status=FAILED -o tests-errors.json

@andrewkroh andrewkroh added the Filebeat Filebeat label May 14, 2020
@andrewkroh
Copy link
Member

run tests

@alakahakai alakahakai added the needs_backport PR is waiting to be backported to other branches. label May 14, 2020
Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please add a changelog entry.

It would be helpful to include some details in the PR description to describe how it addresses the issue.

IIUC what has changed is that the module no longer moves all of the raw fields into the googlecloud.audit namespace, and now is more selective about only copying fields that are mapped. And this prevent any dynamic mapping conflicts. The implication being that some fields that may have been previously copied over are now dropped, but since we have the var.keep_original_message module option this is mitigated. Hopefully I followed correctly 😬 .

@alakahakai
Copy link
Author

Can you please add a changelog entry.

It would be helpful to include some details in the PR description to describe how it addresses the issue.

IIUC what has changed is that the module no longer moves all of the raw fields into the googlecloud.audit namespace, and now is more selective about only copying fields that are mapped. And this prevent any dynamic mapping conflicts. The implication being that some fields that may have been previously copied over are now dropped, but since we have the var.keep_original_message module option this is mitigated. Hopefully I followed correctly 😬 .

Yes, you are correct. The fields have changed recently and these fields don't have defined data models and just used a struct, which caused the issue. The googlecloud.audit dataset now only take in the fields that are defined by the dataset.

@alakahakai alakahakai requested a review from andrewkroh May 14, 2020 22:37
@alakahakai alakahakai merged commit c5bd3e4 into elastic:master May 14, 2020
@alakahakai alakahakai added v7.8.0 and removed needs_backport PR is waiting to be backported to other branches. labels May 14, 2020
alakahakai pushed a commit that referenced this pull request May 15, 2020
 for googlecloud audit log (#18554)

* Add additional fields to address issue #18465 for googlecloud audit log (#18472)
alakahakai pushed a commit that referenced this pull request May 15, 2020
 for googlecloud audit log (#18553)

* Add additional fields to address issue #18465 for googlecloud audit log (#18472)
v1v added a commit to v1v/beats that referenced this pull request May 15, 2020
…w-oss

* upstream/master: (27 commits)
  Disable host fields for "cloud", panw, cef modules (elastic#18223)
  [docs] Rename monitoring collection from legacy internal collection to legacy collection (elastic#18504)
  Introduce auto detection of format (elastic#18095)
  Add additional fields to address issue elastic#18465 for googlecloud audit log (elastic#18472)
  Fix libbeat import path in seccomp policy template (elastic#18418)
  Address Okta input issue elastic#18530 (elastic#18534)
  [Ingest Manager] Avoid Chown on windows (elastic#18512)
  Fix Cisco ASA/FTD msgs that use a host name as NAT address (elastic#18376)
  [CI] Optimise stash/unstash performance (elastic#18473)
  Libbeat: Remove global loggers from libbeat/metric and libbeat/cloudid (elastic#18500)
  Fix PANW bad mapping of client/source and server/dest packets and bytes (elastic#18525)
  Add a file lock to the data directory on startup to prevent multiple agents. (elastic#18483)
  Followup to 12606 (elastic#18316)
  changed input from syslog to tcp/udp due to unsupported RFC (elastic#18447)
  Improve ECS field mappings in Sysmon module. (elastic#18381)
  [Elastic Agent] Cleaner output of inspect command  (elastic#18405)
  [Elastic Agent] Pick up version from libbeat (elastic#18350)
  Update communitybeats.asciidoc (elastic#18470)
  [Metricbeat] Change visualization interval from 15m to >=15m (elastic#18466)
  docs: Fix typo in kerberos docs (elastic#18503)
  ...
alakahakai pushed a commit that referenced this pull request Jun 26, 2020
 for googlecloud audit log (#19470)

* Add additional fields to address issue #18465 for googlecloud audit log (#18472)
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
…sue elastic#18465 for googlecloud audit log (elastic#18554)

* Add additional fields to address issue elastic#18465 for googlecloud audit log (elastic#18472)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants