Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Filebeat] Improve aws-s3 gzip file detection to avoid false negatives #29969

Merged
merged 1 commit into from
Jan 24, 2022
Merged

[Filebeat] Improve aws-s3 gzip file detection to avoid false negatives #29969

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Undo deletion of endpoint config from cloudtrail fileset in {pull}29415[29415]. {pull}29450[29450]
- Make Cisco ASA and FTD modules conform to the ECS definition for event.outcome and event.type. {issue}29581[29581] {pull}29698[29698]
- ibmmq: Fixed `@timestamp` not being populated with correct values. {pull}29773[29773]
- aws-s3: Improve gzip detection to avoid false negatives. {issue}29968[29968]

*Heartbeat*

Expand Down
12 changes: 3 additions & 9 deletions x-pack/filebeat/input/awss3/s3_objects.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ import (
"fmt"
"io"
"io/ioutil"
"net/http"
"reflect"
"strings"
"time"
Expand Down Expand Up @@ -375,18 +374,13 @@ func s3ObjectHash(obj s3EventV2) string {
// stream without consuming it. This makes it convenient for code executed after this function call
// to consume the stream if it wants.
func isStreamGzipped(r *bufio.Reader) (bool, error) {
// Why 512? See https://godoc.org/net/http#DetectContentType
buf, err := r.Peek(512)
buf, err := r.Peek(3)
if err != nil && err != io.EOF {
return false, err
}

switch http.DetectContentType(buf) {
case "application/x-gzip", "application/zip":
return true, nil
default:
return false, nil
}
// gzip magic number (1f 8b) and the compression method (08 for DEFLATE).
return bytes.HasPrefix(buf, []byte{0x1F, 0x8B, 0x08}), nil
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In case this throws false positives, an approach that can be used is to peek enough of the stream to allow the gzip header to be read, an error indicates definitively that the stream is not a gzip. Noting here for future consideration only.

}

// s3Metadata returns a map containing the selected S3 object metadata keys.
Expand Down