Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Gzip compress level 6 #720

Merged
merged 1 commit into from
Aug 9, 2023
Merged

Conversation

pauldraper
Copy link
Contributor

@pauldraper pauldraper commented Aug 4, 2023

Much faster and comparable quality to level 9

In the example below, level 9 (max) compression is 2% smaller and 50% slower compared to level 6. This is why most gzip implementations default to level 6.


BUILD.bazel

load("@rules_pkg//pkg:tar.bzl", "pkg_tar")

pkg_tar(
    name = "tar",
    extension = "tar.gz",
    srcs = ["@src//:src"],
)

WORKSPACE.bazel

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

local_repository(
    name = "rules_pkg",
    path = "../rules_pkg",
)

load("@rules_pkg//:deps.bzl", "rules_pkg_dependencies")

rules_pkg_dependencies()

http_archive(
    build_file_content = """
filegroup(
    name = "src",
    srcs = glob(["**/*"]),
    visibility = ["//visibility:public"],
)
""".strip(),
    name = "src",
    url = "https://github.com/torvalds/linux/archive/c1a515d3c0270628df8ae5f5118ba859b85464a2.zip",
    sha256 = "33e032d6022e19c1b07ff122848b34f2f9e54e3202adb6f14da344b21f01b11a",
)

Level 6: 183MB, 36s

Level 9: 179MB, 54s


(FYI, as a note for anyone caring about performance, using an external gzip binary cut the time by a further 6s.)

Much faster and comparable quality to level 9
@pauldraper pauldraper requested review from aiuto and nacl as code owners August 4, 2023 10:01
Copy link
Collaborator

@aiuto aiuto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know this will come back to haunt me when I try to import it into Google and someone's bloat detector goes off. But I'm feeling lucky.

Expect that in a few weeks I'll come back with a new version that creates a constant for the default at the top of the file, so that it is easy to change for the entire org while vendoring in the code.

@aiuto aiuto merged commit 3ebf3b2 into bazelbuild:main Aug 9, 2023
@pauldraper pauldraper deleted the pauldraper/compress branch August 9, 2023 19:50
flode added a commit to flode/rules_pkg that referenced this pull request Aug 17, 2023
Zip offers different compression algorithms and levels. For packaging,
sometimes faster compression speed is more important than size.
With pkg_tar, there is already the option to use a custom compressor.

For pkg_zip these use case are now possible with the exposed zip
compression_level and compression_type arguments in this PR.

There's a new test case that verifies that the compression settings work by
comparing the compressed file sizes.

The default compression level is the same as recently change for pkg_tar in
bazelbuild#720
flode added a commit to flode/rules_pkg that referenced this pull request Aug 17, 2023
Zip offers different compression algorithms and levels. For packaging,
sometimes faster compression speed is more important than size.
With pkg_tar, there is already the option to use a custom compressor.

For pkg_zip these use case are now possible with the exposed zip
compression_level and compression_type arguments in this PR.

There's a new test case that verifies that the compression settings work by
comparing the compressed file sizes.

The default compression level is the same as recently change for pkg_tar in
bazelbuild#720
flode added a commit to flode/rules_pkg that referenced this pull request Aug 17, 2023
Zip offers different compression algorithms and levels. For packaging,
sometimes faster compression speed is more important than size.
With pkg_tar, there is already the option to use a custom compressor.

For pkg_zip these use case are now possible with the exposed zip
compression_level and compression_type arguments in this PR.

There's a new test case that verifies that the compression settings work by
comparing the compressed file sizes.

The default compression level is the same as recently change for pkg_tar in
bazelbuild#720
flode added a commit to flode/rules_pkg that referenced this pull request Aug 17, 2023
Zip offers different compression algorithms and levels. For packaging,
sometimes faster compression speed is more important than size.
With pkg_tar, there is already the option to use a custom compressor.

For pkg_zip these use case are now possible with the exposed zip
compression_level and compression_type arguments in this PR.

There's a new test case that verifies that the compression settings work by
comparing the compressed file sizes.

The default compression level is the same as recently change for pkg_tar in
bazelbuild#720
alexeagle added a commit that referenced this pull request Aug 17, 2023
Zip offers different compression algorithms and levels. For packaging,
sometimes faster compression speed is more important than size. With
`pkg_tar`, there is already the option to use a custom compressor,
there's no similar option for `pkg_zip` yet.

This PR exposes the zip `compression_level` and `compression_type`
arguments for `pkg_zip`, thus enabling the use case.

The list of compressions is the same as Python's `ZipFile`:
https://docs.python.org/3/library/zipfile.html#zipfile.ZIP_STORED

There's a new test case that verifies that the compression settings work
by comparing the compressed file sizes.

The default compression level is the same as in this recent change for
`pkg_tar` in #720
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants