call for encode br support for brotli compression

[orca-zhang]

I found brotli is more widely used/supported on mobile browsers than gzip these days.

I think, it's about ~23.33%(96.55%-73.22%) delta that is announced.

https://caniuse.com/brotli

https://caniuse.com/mdn-api_compressionstream_compressionstream_gzip

BTW, thanks a ton for your great Caddy, I really like it.

[mholt]

Glad you like Caddy!

Unfortunately, performance of compressing brotli on-the-fly is unacceptable with current implementations. (My understanding is that the brotli format itself is more expensive to encode than decode, so it's possible that it may never be within tolerances.) If it can be made fast enough (with pure Go and/or assembly) then we are open to doing it.

Kind of a duplicate of #3704

Related: #3692

We encourage clients to support Zstandard for more modern on-the-fly compression.

[tasiotas]

Hi @mholt

I totally see your point of Brotli being too slow for on the fly compression, but what about case where Caddy request is being cached? Not having Brotli at all is quite limiting. I'm serving content from 3rd party softwre that does not give me option to precompress assets. Caddy would be perfect place to handle it, and use caching down the pipe.

Thanks

[francislavoie]

There exists a plugin to provide brotli support. You can use that.

But because of the performance concerns, it does not make sense to include in Caddy's standard distribution because users will use it without understanding the performance implications then complain to us when they have very high CPU usage. So we're careful about what we include by default.

[tasiotas]

oh good to hear, thanks!

[mholt]

If you can find us an optimized implementation in Go, we will happily use it. But current implementations grind servers to a halt. Brotli is just hard to encode efficiently.

[sowinski]

Hi, why can you not call from caddy a c library which is doing this faster?

[mohammed90]

Hi, why can you not call from caddy a c library which is doing this faster?

The top reasons are:

• portability: cross-compiling a project with CGO is not fun
• performance: CGO calls are not free, and the compromise in performance has to be truly worth it

Cockroach Labs have a good blogpost about the overhead of using CGO: The cost and complexity of Cgo

[mholt]

Also memory safety. No good if a file can cause a C library to pop out of its memory bounds.

[dunglas]

For the record, I created a module that relies on the official C implementation (and the Google Go module): https://github.com/dunglas/caddy-cbrotli

[sowinski]

Is this implementation fast enough to run it for dynamic created content or is it only good for static files?

[dunglas]

@sowinski according to my tests it is good enough dynamic content.

[sowinski]

would be nice if this feature would fine the way to the next caddy version

[mohammed90]

would be nice if this feature would fine the way to the next caddy version

Kevin's plugin still uses C (CGO). There's near zero chance we're admitting C dependency into Caddy. Plugins can be developed and built for users who desire to accept the risk and complexity of having C in the stack.