Transfer-codings

[mnot]

The current draft does not accommodate transfer-codings. There has been some pushback at this, since in some situations (e.g., range requests), it is difficult to compress without them.

In discussion, it appears unlikely that we'd require support for transfer-codings; however, it might be that we want to accommodate them, or a subset.

One simple remedy might be to add a flag on DATA indicating that the body is compressed, and have a corresponding setting to negotiate its use.

[cben]

historical notes:
[I hoped to use TE compression without CE in a place where CE negotiation is messy, so was sad TE is gone in HTTP/2, and got curious why...]

Result

The per-frame compression added in #463 was later reverted in #511 due to concerns #497.

=> HTTP/2 has only HPACK headers and CE content compression, as does https://tools.ietf.org/html/draft-ietf-quic-http-17.

initial state:

• initial "no TE compression" decision somewhat lost to time; rumors I read are (1) desire to get rid of hop-by-hop headers (2) SPDY experience not supporting it.

• "CE gzip was made a fixed part of the protocol, in part, because intercepting intermediaries (or antivirus libraries as a subset of that) were explicitly stripping negotiation for it for their own convenience. Making it non negotiable enhances the robustness of the protocol against that which is imo the right trade given the history here and the importance of compression."

security

#423 and other security concerns mentioned several times thoughout:

• BREACH attack was known and future problems were conceivable.
  • a recurring point is that only origin server knows when compression is safe, thus proxies adding hop-by-hop compression are risky; a recurring rebuttal is proxies already do (incorrectly) add hop-by-hop CE.
  • AFAICT there was concensus that "don't add compression" and "dont merge" make TE safe (if obeyed).
• DoS using decompressing intermediary to amplify
• forcing the other end to allocate large memory

discussions summary

Obviously subjective; in particular I understand servers & clients well but intermediaries badly.

omitting most discussion on #424 which is about CE/AE, except where relevant.

Email thread on reviving TE compression: https://lists.w3.org/Archives/Public/ietf-http-wg/2014JanMar/thread.html#msg1179

• A concrete proposal by Keith Morgan & Christoph Brunhuber to add TE and somewhat deprecate CE compression here: https://lists.w3.org/Archives/Public/ietf-http-wg/2014JanMar/1205.html
  (this got pushback about 2-to-1.x translating intermediaries and about CE compression having its place when server already has both representations)

• near the end, Matthew Kerwin suggests having effectively removed hop-by-hop headers may have good effects (or "even just
  clarity of the spec") and floats idea of new frame type

Next thread: https://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/thread.html#msg59

• Matthew Kerwin proposed new optional ENCODING field in DATA frames, with SETTINGS negotiation.
• https://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0156.html
  experimental data on this being per-frame: compresses worse than whole-stream but surprisingly not much worse, and has benefits (less state)
• https://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0076.html Keith Morgan compares to TE and proposes 2 retains 1.1 TE semantics minus "chunked".
• https://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0103.html Roberto Peon says transport compression in SPDY didn't work well
• https://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0113.html Keith Morgan sumarizes goals and problems so far; proposes "Message-Encoding" that is end-to-end like CE but encoded after Range is applied.

"Porting T-E to HTTP/2: Reasons Against" thread https://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/thread.html#msg166

"Transfer-codings, mandatory content-coding support and intermediaries" thread https://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/thread.html#msg273

• Requiring Clients to support content-codings #460 implicitly assumed AE gzip breaks semantic transparency of proxies (this was also discussed before first TE thread) => led to dropping implicitly assumed AE gzip.
• several people point out TE would solve the goal of semantically transparent compression better.
• discussion whether security concern is solved by forbidding intermediaries to add compression.

"#445: Transfer-Codings" thread https://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/thread.html#msg442

• proposed GZIP in DATA #463 (per-frame gzip bit) which got in!
• fine-tuning of Security Considerations, esp. "don't merge".
• discussion of per-frame vs per-segment => Compress segments rather than frames #466

"#466 segment compression" thread https://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/thread.html#msg523

• per-frame doesn't require retaining state
• per-segment simpler for proxies: "since an intermediary already can’t re-frame across segment boundaries"
• DoS / memory / multiplexing tradeoff discussions...
• => closed "per-frame is good enough, and the DoS considerations are too much of a worry".

"Making Implicit C-E work" thread. https://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/thread.html#msg529

• distinguishing implicit CE from explicit CE — basically shot down as complex & modifying CE semantics out of charter for HTTP/2.
• much bemoaning of CE, some mentioning of TE but no concrete progress.
• much DoS discussion.
• Message-Encoding spun off as Draft RFC!
  https://github.com/shearl/Internet-Drafts/blob/master/draft-morgan-http-message-encoding/draft-morgan-http-message-encoding.txt
  https://tools.ietf.org/html/draft-morgan-http2-header-compression-00

"Range Requests vs Content Codings #msg1327" thread https://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/thread.html#msg1327

• proposes a new range unit, "bbcc" (bytes-before-content-coding)
• didn't fully read; this is not about TE but attempt to solve some of the problems with CE.

I don't see email discussion that led to #511 except for the brief motivation in #497.
(Decided in NYC meeting; agenda & minutes etherpad are down by now.)

My impressions (very subjective, bias: I love TE)

• it was not for lack of good will 👼
• TE would easily get in if there was no security considerations shadow over the whole area 🔐
• HTTP/1 interoperability and proxies gave status quo a strong advantage — even despite CE's interoperability and proxies problems :-( Can't blame a standards process for this!
• The preference for frame/segment compression over just allowing TE gzip wasn't suffiently motivated?
  Eventually frame-vs-segment proved a problematic tradeoff but TE wasn't resurrected.
• it seems participants have a blind spot for CE: people frequently called out problems with TE that equally apply to CE! But this was swiftly called out, so can't say if it influenced the process.