Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add compression support #86

Merged
merged 3 commits into from
Aug 12, 2020
Merged

Add compression support #86

merged 3 commits into from
Aug 12, 2020

Conversation

camdencheek
Copy link
Contributor

@camdencheek camdencheek commented Aug 11, 2020

Description of Changes

Adds compression support to the Google Cloud Output.

Also increases the timeout for write calls and reduces the handler limit to reduce contention for network resources.

Please check that the PR fulfills these requirements

  • Tests for the changes have been added (for bug fixes / features)
  • Docs have been added / updated (for bug fixes / features)
  • Add a changelog entry (for non-trivial bug fixes / features)
  • CI passes

@codecov
Copy link

codecov bot commented Aug 11, 2020

Codecov Report

Merging #86 into master will decrease coverage by 0.02%.
The diff coverage is 75.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #86      +/-   ##
==========================================
- Coverage   74.14%   74.12%   -0.02%     
==========================================
  Files          67       67              
  Lines        4045     4049       +4     
==========================================
+ Hits         2999     3001       +2     
- Misses        790      793       +3     
+ Partials      256      255       -1     
Impacted Files Coverage Δ
operator/builtin/output/google_cloud.go 41.81% <71.43%> (+0.14%) ⬆️
operator/buffer/buffer.go 80.00% <100.00%> (ø)
operator/builtin/input/file/read_to_end.go 59.70% <0.00%> (-2.99%) ⬇️
operator/builtin/input/tcp.go 79.17% <0.00%> (+2.78%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 478054e...65387fa. Read the comment docs.

@djaglowski
Copy link
Member

Log Files Logs / Second CPU Avg (%) CPU Avg Δ (%) Memory Avg (MB) Memory Avg Δ (MB)
1 1000 2.9828258 -0.44835377 27.034887 -1.5230331
1 5000 7.7415905 -0.4310627 41.65019 +1.5343475
1 10000 14.603746 +0.24137211 53.191944 -0.25444412
1 50000 56.5999 -1.4825096 194.21013 +24.897629
1 100000 108.86462 +0.31905365 296.98978 +13.574646
10 100 4.72421 -0.034507275 27.45474 -0.2385521
10 500 9.845065 -0.120625496 38.44639 +0.9277992
10 1000 16.879932 +0.65540314 50.616512 -1.8305511
10 5000 63.896656 +0.05116272 190.16743 +23.185211
10 10000 122.18321 +5.5409393 333.67053 +16.24057

@camdencheek camdencheek requested a review from djaglowski August 11, 2020 16:36
@djaglowski
Copy link
Member

Log Files Logs / Second CPU Avg (%) CPU Avg Δ (%) Memory Avg (MB) Memory Avg Δ (MB)
1 1000 3.1207786 -0.13792253 28.138874 +0.90153503
1 5000 7.7932086 +0.15513277 39.49313 -1.3228722
1 10000 13.965798 +0.08614349 52.639816 -1.396553
1 50000 60.9853 -1.2012253 187.88321 -3.8834991
1 100000 109.72072 -0.7491455 272.99274 +7.5849915
10 100 5.7070365 +1.4483466 29.130522 -0.20770645
10 500 10.120911 +0.05175495 40.652073 +0.5786629
10 1000 16.414215 +0.17245865 50.505928 -1.4353447
10 5000 65.1638 +2.8003273 163.68346 -18.401001
10 10000 117.101555 +1.3749237 299.5621 -30.465637

@camdencheek camdencheek merged commit 1351568 into master Aug 12, 2020
@camdencheek camdencheek deleted the compress branch August 12, 2020 21:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants