Filebeat gcs input addFiledJobs panic protection #38407

adassow · 2024-03-19T07:18:23Z

In case obj, err := s.bucket.Object(name).Attrs(ctx) return an error, obj will be probably nil in consequence the code below will panic
objectURI := "gs://" + s.src.BucketName + "/" + obj.Name

Proposed commit message

Filebeat gcs input addFiledJobs panic protection

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

No idea

How to test this PR locally

No idea

Related issues

No idea

Use cases

No idea

Screenshots

Logs

Input crashed with: input panic with: runtime error: invalid memory address or nil pointer dereference
goroutine 938 [running]:
runtime/debug.Stack()
	runtime/debug/stack.go:24 +0x65
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*managedInput).runSource.func1()
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/input.go:143 +0x5d
panic({0x558d0f92f8a0, 0x558d123846f0})
	runtime/panic.go:884 +0x213
github.com/elastic/beats/v7/x-pack/filebeat/input/gcs.(*scheduler).addFailedJobs(0xc002e6cb40, {0x558d1007a820, 0xc002e720a0}, {0xc0042de240, 0x3, 0x4})
	github.com/elastic/beats/v7/x-pack/filebeat/input/gcs/scheduler.go:238 +0x378
github.com/elastic/beats/v7/x-pack/filebeat/input/gcs.(*scheduler).scheduleOnce(0xc002e6cb40, {0x558d1007a820?, 0xc002e720a0})
	github.com/elastic/beats/v7/x-pack/filebeat/input/gcs/scheduler.go:106 +0x2d3
github.com/elastic/beats/v7/x-pack/filebeat/input/gcs.(*scheduler).schedule(0xc002e6cb40, {0x558d1007a820?, 0xc002e720a0})
	github.com/elastic/beats/v7/x-pack/filebeat/input/gcs/scheduler.go:60 +0x65
github.com/elastic/beats/v7/x-pack/filebeat/input/gcs.(*gcsInput).Run(_, {0xc006f6a490, {0xc002e70000, 0x86}, {{0x558d0eae5cfe, 0x8}, {0x558d0eae5cfe, 0x8}, {0x558d0eadf24e, 0x6}, ...}, ...}, ...)
	github.com/elastic/beats/v7/x-pack/filebeat/input/gcs/input.go:183 +0x885
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*managedInput).runSource(_, {0xc006f6a490, {0xc002e70000, 0x86}, {{0x558d0eae5cfe, 0x8}, {0x558d0eae5cfe, 0x8}, {0x558d0eadf24e, 0x6}, ...}, ...}, ...)
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/input.go:168 +0x479
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*managedInput).Run.func1()
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/input.go:122 +0x1bf
github.com/elastic/go-concert/unison.(*MultiErrGroup).Go.func1()
	github.com/elastic/[email protected]/unison/multierrgroup.go:42 +0x75
created by github.com/elastic/go-concert/unison.(*MultiErrGroup).Go
	github.com/elastic/[email protected]/unison/multierrgroup.go:40 +0x8d

In case `obj, err := s.bucket.Object(name).Attrs(ctx)` return an error, obj will be probably `nil` in consequence the code below will panic `objectURI := "gs://" + s.src.BucketName + "/" + obj.Name`

cla-checker-service · 2024-03-19T07:18:28Z

💚 CLA has been signed

mergify · 2024-03-19T07:19:00Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @adassow? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

elasticmachine · 2024-03-19T07:23:54Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Duration: 137 min 43 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

adassow · 2024-03-19T08:21:52Z

This pull request does not have a backport label. If this is a bug or security fix, could you label this PR @adassow? 🙏. For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack

The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

It seems I don't have permission to add any label. It's a bugfix and doesn't change anything except protecting against panic. It shout go probably as a patch.

ShourieG · 2024-03-19T08:26:53Z

/test

elasticmachine · 2024-03-19T08:27:54Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

ShourieG · 2024-03-19T08:52:46Z

Hi @adassow, thanks for the PR, will get it merged asap. Could you please add an entry in CHANGELOG-developer.next.asciidoc, that would be great.

ShourieG

LGTM

ShourieG

Lgtm

adassow · 2024-03-20T07:54:50Z

@ShourieG How can I satisfy buildkite/docs-build-pr check?

ShourieG · 2024-03-20T08:20:59Z

@ShourieG How can I satisfy buildkite/docs-build-pr check?

@adassow There seems to be a CI issue on our end, the CI team is looking into it. As soon as it's resolved we'll merge.

karenzone · 2024-03-20T12:39:10Z

buildkite test this rebuild

andrewkroh · 2024-03-20T12:53:15Z

CHANGELOG-developer.next.asciidoc

@@ -92,6 +92,7 @@ The list below covers the major changes between 7.0.0-rc2 and main only.
 - Make winlogbeat/sys/wineventlog follow the unsafe.Pointer rules. {pull}36650[36650]
 - Cleaned up documentation errors & fixed a minor bug in Filebeat Azure blob storage input. {pull}36714[36714]
 - Fix copy arguments for strict aligned architectures. {pull}36976[36976]
+- Fix filebit gcs input panic {pull}38407[38407]


This entry should be added to CHANGELOG.next.asciidoc. The "developer" file is focused mainly on API changes or internal-only fixes (without a user impact).

filebit -> filebeat

I will create a PR to shift this around, since auto merge was enabled and it got merged.

* Filebeat gcs input addFiledJobs panic protection In case `obj, err := s.bucket.Object(name).Attrs(ctx)` return an error, obj will be probably `nil` in consequence the code below will panic `objectURI := "gs://" + s.src.BucketName + "/" + obj.Name` * Update CHANGELOG-developer.next.asciidoc (cherry picked from commit c6fd99c)

…ion (#38475) * Filebeat gcs input addFiledJobs panic protection (#38407) * Filebeat gcs input addFiledJobs panic protection In case `obj, err := s.bucket.Object(name).Attrs(ctx)` return an error, obj will be probably `nil` in consequence the code below will panic `objectURI := "gs://" + s.src.BucketName + "/" + obj.Name` * Update CHANGELOG-developer.next.asciidoc (cherry picked from commit c6fd99c) * Update CHANGELOG-developer.next.asciidoc --------- Co-authored-by: Adam Sowiński <[email protected]> Co-authored-by: ShourieG <[email protected]>

…ion (#38474) * Filebeat gcs input addFiledJobs panic protection (#38407) * Filebeat gcs input addFiledJobs panic protection In case `obj, err := s.bucket.Object(name).Attrs(ctx)` return an error, obj will be probably `nil` in consequence the code below will panic `objectURI := "gs://" + s.src.BucketName + "/" + obj.Name` * Update CHANGELOG-developer.next.asciidoc (cherry picked from commit c6fd99c) * Update CHANGELOG-developer.next.asciidoc --------- Co-authored-by: Adam Sowiński <[email protected]> Co-authored-by: ShourieG <[email protected]>

Filebeat gcs input addFiledJobs panic protection

171414c

In case `obj, err := s.bucket.Object(name).Attrs(ctx)` return an error, obj will be probably `nil` in consequence the code below will panic `objectURI := "gs://" + s.src.BucketName + "/" + obj.Name`

adassow requested a review from a team as a code owner March 19, 2024 07:18

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Mar 19, 2024

mergify bot assigned adassow Mar 19, 2024

adassow removed their assignment Mar 19, 2024

mergify bot assigned adassow Mar 19, 2024

ShourieG added bugfix 8.13-candidate labels Mar 19, 2024

ShourieG added the Team:Security-Service Integrations Security Service Integrations Team label Mar 19, 2024

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Mar 19, 2024

ShourieG added needs_team Indicates that the issue/PR needs a Team:* label backport-v8.12.0 Automated backport with mergify labels Mar 19, 2024

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Mar 19, 2024

ShourieG added the backport-v8.13.0 Automated backport with mergify label Mar 19, 2024

ShourieG approved these changes Mar 19, 2024

View reviewed changes

ShourieG enabled auto-merge (squash) March 19, 2024 10:28

Update CHANGELOG-developer.next.asciidoc

26609d1

auto-merge was automatically disabled March 19, 2024 11:04
Head branch was pushed to by a user without write access

ShourieG approved these changes Mar 19, 2024

View reviewed changes

ShourieG enabled auto-merge (squash) March 19, 2024 12:36

ShourieG disabled auto-merge March 19, 2024 14:33

ShourieG enabled auto-merge (squash) March 19, 2024 14:35

Merge branch 'main' into patch-1

193054b

andrewkroh reviewed Mar 20, 2024

View reviewed changes

ShourieG merged commit c6fd99c into elastic:main Mar 20, 2024
22 checks passed

This was referenced Mar 20, 2024

[8.12](backport #38407) Filebeat gcs input addFiledJobs panic protection #38474

Merged

[8.13](backport #38407) Filebeat gcs input addFiledJobs panic protection #38475

Merged

ShourieG mentioned this pull request Mar 21, 2024

[filebeat][gcs] - Cleaned up invalid CHANGELOG entry resulting from community contribution #38512

Merged

6 tasks

mergify bot mentioned this pull request Mar 21, 2024

[8.13](backport #38512) [filebeat][gcs] - Cleaned up invalid CHANGELOG entry resulting from community contribution #38516

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Filebeat gcs input addFiledJobs panic protection #38407

Filebeat gcs input addFiledJobs panic protection #38407

adassow commented Mar 19, 2024

cla-checker-service bot commented Mar 19, 2024 •

edited

Loading

mergify bot commented Mar 19, 2024

elasticmachine commented Mar 19, 2024 •

edited

Loading

Build stats

adassow commented Mar 19, 2024

ShourieG commented Mar 19, 2024

elasticmachine commented Mar 19, 2024

ShourieG commented Mar 19, 2024 •

edited

Loading

ShourieG left a comment

ShourieG left a comment

adassow commented Mar 20, 2024

ShourieG commented Mar 20, 2024

karenzone commented Mar 20, 2024

andrewkroh Mar 20, 2024

ShourieG Mar 21, 2024

ShourieG Mar 21, 2024

Filebeat gcs input addFiledJobs panic protection #38407

Filebeat gcs input addFiledJobs panic protection #38407

Conversation

adassow commented Mar 19, 2024

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

cla-checker-service bot commented Mar 19, 2024 • edited Loading

mergify bot commented Mar 19, 2024

elasticmachine commented Mar 19, 2024 • edited Loading

💚 Build Succeeded

Build stats

❕ Flaky test report

🤖 GitHub comments

adassow commented Mar 19, 2024

ShourieG commented Mar 19, 2024

elasticmachine commented Mar 19, 2024

ShourieG commented Mar 19, 2024 • edited Loading

ShourieG left a comment

Choose a reason for hiding this comment

ShourieG left a comment

Choose a reason for hiding this comment

adassow commented Mar 20, 2024

ShourieG commented Mar 20, 2024

karenzone commented Mar 20, 2024

andrewkroh Mar 20, 2024

Choose a reason for hiding this comment

ShourieG Mar 21, 2024

Choose a reason for hiding this comment

ShourieG Mar 21, 2024

Choose a reason for hiding this comment

cla-checker-service bot commented Mar 19, 2024 •

edited

Loading

elasticmachine commented Mar 19, 2024 •

edited

Loading

ShourieG commented Mar 19, 2024 •

edited

Loading