Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decompress S3 files without ".gz" extension #18283

Closed
fdartayre opened this issue May 6, 2020 · 3 comments · Fixed by #18764
Closed

Decompress S3 files without ".gz" extension #18283

fdartayre opened this issue May 6, 2020 · 3 comments · Fixed by #18764
Assignees
Labels
enhancement Filebeat Filebeat Team:Platforms Label for the Integrations - Platforms team

Comments

@fdartayre
Copy link
Contributor

Describe a specific use case for the enhancement or feature:
Some files automatically generated by AWS are gzip compressed but named without a .gz extension.

Describe the enhancement:
Support for gzipped files in S3 input has been introduced in 7.5.0 (#13980) but as it relies on the .gz extension of the files, the S3 input doesn't decompress the files with a .gz extension. The request is to be able to decompress independently of the naming.

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label May 6, 2020
@andresrc andresrc added the Team:Platforms Label for the Integrations - Platforms team label May 6, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-platforms (Team:Platforms)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label May 6, 2020
@exekias
Copy link
Contributor

exekias commented May 6, 2020

I think this makes sense, we could do something like this:

Allow to configure a list of regexps to determine if decompression must be done, the default would be just \.gz$

@kaiyan-sheng
Copy link
Contributor

Thank you @fdartayre for creating this issue!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Filebeat Filebeat Team:Platforms Label for the Integrations - Platforms team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants