Response timeout when an h2c request was sent and the response's content-encoding is gzip

[qingdaoheze]

Jetty version
9.4.19.v20190610

Java version
1.8

OS type/version
MAC OS / 10.15.6

Description
Use the following code to send a h2c request to a tomcat server(tomcat-embeded-core-9.0.21), when the response's content-encoding is gzip, the client will be blocked until timeout.

        HTTP2Client http2Client = new HTTP2Client();
        http2Client.setSelectors(1);
        HttpClientTransport transport = new HttpClientTransportOverHTTP2(http2Client);

        HttpClient httpClient = new HttpClient(transport, null);
        httpClient.setConnectTimeout(1000);
        httpClient.setIdleTimeout(5000);
        httpClient.setFollowRedirects(true);
        httpClient.start();

        try {
            ContentProvider contentProvider = new StringContentProvider("application/json", content, StandardCharsets.UTF_8);
            Request request = httpClient.POST(url).content(contentProvider, "application/json");
            request.timeout(30, TimeUnit.SECONDS);
            request.idleTimeout(3, TimeUnit.SECONDS);
            ContentResponse contentResponse = request.send();
            int status = contentResponse.getStatus();
            log.info("status:{}", status);
            HttpVersion version = contentResponse.getVersion();
            log.info("version: {}", version);
            System.out.println(version);
            String contentAsString = contentResponse.getContentAsString();
            System.out.println(contentAsString);
            log.info("response:{}", contentAsString);
            String contentEncoding = contentResponse.getHeaders().get(HttpHeader.CONTENT_ENCODING);
            log.info("contentEncoding:{}", contentEncoding);
        } finally {
            httpClient.stop();
        }

[qingdaoheze]

9.4.39.v20210325 and 10.0.2 have the same problem.

[sbordet]

Please upgrade to the latest Jetty.

Does the server support (or is configured with) prior-knowledge clear-text HTTP/2?

Can you run the above with DEBUG logging, collect the logs in a file and attach the log file to this issue?

[qingdaoheze]

The logs as below:

22:48:28.270 [main] DEBUG org.eclipse.jetty.client.HttpClient - Created HttpDestination[http://127.0.0.1:7004]@7f13d6e,queue=0,pool=MultiplexConnectionPool@51cdd8a[c=0/64,b=0,m=0,i=0]
22:48:28.271 [main] DEBUG o.e.jetty.client.HttpConversation - Exchanges in conversation 1, override=null, listeners=[org.eclipse.jetty.client.util.FutureResponseListener@48503868]
22:48:28.274 [main] DEBUG o.e.jetty.client.HttpDestination - org.eclipse.jetty.client.HttpDestination$TimeoutTask@53f65459 scheduled timeout in 5988 ms
22:48:28.274 [main] DEBUG o.e.jetty.client.HttpDestination - Queued HttpRequest[POST /big-body HTTP/1.1]@3b088d51 for HttpDestination[http://127.0.0.1:7004]@7f13d6e,queue=1,pool=MultiplexConnectionPool@51cdd8a[c=0/64,b=0,m=0,i=0]
22:48:28.275 [main] DEBUG o.e.j.client.AbstractConnectionPool - tryCreate 0/64 connections 0/1 pending
22:48:28.275 [main] DEBUG o.e.j.client.AbstractConnectionPool - newConnection 1/64 connections 1/1 pending
22:48:28.357 [HttpClient@68be2bc2-13] DEBUG o.e.j.client.AbstractConnectionPool - Connection 1/64 creation succeeded HttpConnectionOverHTTP2@7cb68078(closed=false)[HTTP2ClientSession@20fef01c{l:/127.0.0.1:55074 <-> r:/127.0.0.1:7004,sendWindow=65535,recvWindow=16777216,streams=0,NOT_CLOSED,null}]
22:48:28.357 [HttpClient@68be2bc2-13] DEBUG o.e.j.client.AbstractConnectionPool - Connection idle HttpConnectionOverHTTP2@7cb68078(closed=false)[HTTP2ClientSession@20fef01c{l:/127.0.0.1:55074 <-> r:/127.0.0.1:7004,sendWindow=65535,recvWindow=16777216,streams=0,NOT_CLOSED,null}]
22:48:28.358 [HttpClient@68be2bc2-13] DEBUG o.e.j.client.AbstractConnectionPool - Connection active HttpConnectionOverHTTP2@7cb68078(closed=false)[HTTP2ClientSession@20fef01c{l:/127.0.0.1:55074 <-> r:/127.0.0.1:7004,sendWindow=65535,recvWindow=16777216,streams=0,NOT_CLOSED,null}]
22:48:28.358 [HttpClient@68be2bc2-13] DEBUG o.e.jetty.client.HttpDestination - Processing exchange HttpExchange@6ce62fb9 req=PENDING/null@null res=PENDING/null@null on HttpConnectionOverHTTP2@7cb68078(closed=false)[HTTP2ClientSession@20fef01c{l:/127.0.0.1:55074 <-> r:/127.0.0.1:7004,sendWindow=65535,recvWindow=16777216,streams=0,NOT_CLOSED,null}] of HttpDestination[http://127.0.0.1:7004]@7f13d6e,queue=0,pool=MultiplexConnectionPool@51cdd8a[c=1/64,b=0,m=1,i=0]
22:48:28.365 [HttpClient@68be2bc2-13] DEBUG org.eclipse.jetty.client.HttpChannel - HttpExchange@6ce62fb9 req=PENDING/null@null res=PENDING/null@null associated true to HttpChannelOverHTTP2@76497e77(exchange=HttpExchange@6ce62fb9 req=PENDING/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP2@37cd9630(req=QUEUED,snd=IDLE,failure=null),recv=HttpReceiverOverHTTP2@3e32101a(rsp=IDLE,failure=null)]
22:48:28.365 [HttpClient@68be2bc2-13] DEBUG o.e.j.client.TimeoutCompleteListener - Scheduled timeout in 5895 ms for HttpRequest[POST /big-body HTTP/2.0]@3b088d51 on org.eclipse.jetty.client.TimeoutCompleteListener@4e69d6e3
22:48:28.365 [HttpClient@68be2bc2-13] DEBUG org.eclipse.jetty.client.HttpSender - Request begin HttpRequest[POST /big-body HTTP/2.0]@3b088d51
22:48:28.367 [HttpClient@68be2bc2-13] DEBUG org.eclipse.jetty.client.HttpSender - Request headers HttpRequest[POST /big-body HTTP/2.0]@3b088d51
Accept-Encoding: gzip
User-Agent: Jetty/9.4.19.v20190610
Content-Type: application/json
Content-Length: 0
22:48:28.373 [HttpClient@68be2bc2-13] DEBUG org.eclipse.jetty.client.HttpContent - Advanced content to last chunk java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]
22:48:28.375 [HttpClient@68be2bc2-13] DEBUG org.eclipse.jetty.client.HttpSender - Request committed HttpRequest[POST /big-body HTTP/2.0]@3b088d51
22:48:28.375 [HttpClient@68be2bc2-13] DEBUG org.eclipse.jetty.client.HttpSender - Request content HttpRequest[POST /big-body HTTP/2.0]@3b088d51
HeapByteBuffer@5d0b3572[p=0,l=0,c=0,r=0]={<<<>>>}
22:48:28.375 [HttpClient@68be2bc2-13] DEBUG org.eclipse.jetty.client.HttpContent - Advanced content past last chunk
22:48:28.375 [HttpClient@68be2bc2-13] DEBUG org.eclipse.jetty.client.HttpSender - Content present false, last true, consumed true for HttpRequest[POST /big-body HTTP/2.0]@3b088d51
22:48:28.376 [HttpClient@68be2bc2-13] DEBUG org.eclipse.jetty.client.HttpSender - Request success HttpRequest[POST /big-body HTTP/2.0]@3b088d51
22:48:28.376 [HttpClient@68be2bc2-13] DEBUG o.eclipse.jetty.client.HttpExchange - Terminated request for HttpExchange@6ce62fb9 req=TERMINATED/null@null res=PENDING/null@null, result: null
22:48:28.376 [HttpClient@68be2bc2-13] DEBUG org.eclipse.jetty.client.HttpSender - Terminating request HttpRequest[POST /big-body HTTP/2.0]@3b088d51
22:48:28.386 [HttpClient@68be2bc2-11] DEBUG o.e.jetty.client.HttpConversation - Exchanges in conversation 1, override=null, listeners=[org.eclipse.jetty.client.util.FutureResponseListener@48503868, org.eclipse.jetty.client.TimeoutCompleteListener@4e69d6e3]
22:48:28.387 [HttpClient@68be2bc2-11] DEBUG o.eclipse.jetty.client.HttpReceiver - Response begin HttpResponse[HTTP/2.0 200 null]@2d95c30d
22:48:28.387 [HttpClient@68be2bc2-11] DEBUG o.eclipse.jetty.client.HttpReceiver - Response headers HttpResponse[HTTP/2.0 200 null]@2d95c30d
content-type: application/json;charset=UTF-8
content-length: 19781
date: Tue, 06 Apr 2021 14:48:28 GMT
vary: accept-encoding
content-encoding: gzip
22:48:28.394 [HttpClient@68be2bc2-11] DEBUG o.eclipse.jetty.client.HttpReceiver - Response content HttpResponse[HTTP/2.0 200 null]@2d95c30d
HeapByteBuffer@14886320[p=0,l=4525,c=4525,r=4525]={<<<\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00L\x9cM\xAe\xEc\xBc\x91...\xB9v\x92\x10\xE3\xCf\x9f\xFf\xFe\x7f\x00\x00\x00\xFf\xFf>>>}
22:48:28.395 [HttpClient@68be2bc2-11] DEBUG o.eclipse.jetty.client.HttpReceiver - Response content decoded (org.eclipse.jetty.client.GZIPContentDecoder@2c77d7eb) HttpResponse[HTTP/2.0 200 null]@2d95c30d
HeapByteBuffer@640a0f3c[p=0,l=8192,c=8192,r=8192]={<<<{"key297":"value2...81","key580":"v>>>}
22:48:28.395 [HttpClient@68be2bc2-11] DEBUG o.eclipse.jetty.client.HttpReceiver - Response content decoded (org.eclipse.jetty.client.GZIPContentDecoder@2c77d7eb) HttpResponse[HTTP/2.0 200 null]@2d95c30d
HeapByteBuffer@640a0f3c[p=0,l=8192,c=8192,r=8192]={<<<alue580","key158"...39","key38":"va>>>}
22:48:28.395 [HttpClient@68be2bc2-11] DEBUG o.eclipse.jetty.client.HttpReceiver - Response content decoded (org.eclipse.jetty.client.GZIPContentDecoder@2c77d7eb) HttpResponse[HTTP/2.0 200 null]@2d95c30d
HeapByteBuffer@640a0f3c[p=0,l=3397,c=8192,r=3397]={<<<lue38","key249":"...61":"value261"}>>>:"value635","key8...39","key38":"va}
22:48:28.397 [HttpClient@68be2bc2-11] DEBUG o.e.jetty.client.HttpConnection - HTTP2Stream@37f1f84a#1{sendWindow=65535,recvWindow=8384073,reset=true/false,CLOSED,age=27,attachment=HttpChannelOverHTTP2@76497e77(exchange=HttpExchange@6ce62fb9 req=TERMINATED/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP2@37cd9630(req=QUEUED,snd=COMPLETED,failure=null),recv=HttpReceiverOverHTTP2@3e32101a(rsp=CONTENT,failure=null)]} closed for HttpChannelOverHTTP2@76497e77(exchange=HttpExchange@6ce62fb9 req=TERMINATED/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP2@37cd9630(req=QUEUED,snd=COMPLETED,failure=null),recv=HttpReceiverOverHTTP2@3e32101a(rsp=CONTENT,failure=null)]
22:48:28.397 [HttpClient@68be2bc2-11] DEBUG o.e.jetty.client.HttpDestination - Released HttpConnectionOverHTTP2@7cb68078(closed=false)[HTTP2ClientSession@20fef01c{l:/127.0.0.1:55074 <-> r:/127.0.0.1:7004,sendWindow=65535,recvWindow=16772681,streams=1,NOT_CLOSED,null}]
22:48:28.397 [HttpClient@68be2bc2-11] DEBUG o.e.j.client.AbstractConnectionPool - Connection idle HttpConnectionOverHTTP2@7cb68078(closed=false)[HTTP2ClientSession@20fef01c{l:/127.0.0.1:55074 <-> r:/127.0.0.1:7004,sendWindow=65535,recvWindow=16772681,streams=1,NOT_CLOSED,null}]
22:48:34.265 [HttpClient@68be2bc2-scheduler] DEBUG o.e.j.client.TimeoutCompleteListener - Total timeout 6000 ms elapsed for HttpRequest[POST /big-body HTTP/2.0]@3b088d51 on org.eclipse.jetty.client.TimeoutCompleteListener@4e69d6e3
22:48:34.266 [HttpClient@68be2bc2-scheduler] DEBUG o.eclipse.jetty.client.HttpExchange - Failed HttpExchange@6ce62fb9 req=TERMINATED/null@null res=COMPLETED/java.util.concurrent.TimeoutException: Total timeout 6000 ms elapsed@3ff6c3b1: req=false/rsp=true {}
22:48:34.266 [HttpClient@68be2bc2-scheduler] DEBUG o.eclipse.jetty.client.HttpReceiver - Response failure HttpResponse[HTTP/2.0 200 null]@2d95c30d HttpExchange@6ce62fb9 req=TERMINATED/null@null res=COMPLETED/java.util.concurrent.TimeoutException: Total timeout 6000 ms elapsed@3ff6c3b1 on HttpChannelOverHTTP2@76497e77(exchange=HttpExchange@6ce62fb9 req=TERMINATED/null@null res=COMPLETED/java.util.concurrent.TimeoutException: Total timeout 6000 ms elapsed@3ff6c3b1)[send=HttpSenderOverHTTP2@37cd9630(req=QUEUED,snd=COMPLETED,failure=null),recv=HttpReceiverOverHTTP2@3e32101a(rsp=FAILURE,failure=java.util.concurrent.TimeoutException: Total timeout 6000 ms elapsed)]: {}
22:48:34.267 [HttpClient@68be2bc2-scheduler] DEBUG o.eclipse.jetty.client.HttpExchange - Terminated response for HttpExchange@6ce62fb9 req=TERMINATED/null@null res=TERMINATED/java.util.concurrent.TimeoutException: Total timeout 6000 ms elapsed@3ff6c3b1, result: Result[HttpRequest[POST /big-body HTTP/2.0]@3b088d51 > HttpResponse[HTTP/2.0 200 null]@2d95c30d] java.util.concurrent.TimeoutException: Total timeout 6000 ms elapsed
22:48:34.268 [HttpClient@68be2bc2-scheduler] DEBUG o.eclipse.jetty.client.HttpReceiver - Response complete HttpResponse[HTTP/2.0 200 null]@2d95c30d
22:48:34.268 [HttpClient@68be2bc2-scheduler] DEBUG org.eclipse.jetty.client.HttpChannel - HttpExchange@6ce62fb9 req=TERMINATED/null@null res=TERMINATED/java.util.concurrent.TimeoutException: Total timeout 6000 ms elapsed@3ff6c3b1 disassociated true from HttpChannelOverHTTP2@76497e77(exchange=null)[send=HttpSenderOverHTTP2@37cd9630(req=QUEUED,snd=COMPLETED,failure=null),recv=HttpReceiverOverHTTP2@3e32101a(rsp=FAILURE,failure=java.util.concurrent.TimeoutException: Total timeout 6000 ms elapsed)]
22:48:34.268 [HttpClient@68be2bc2-scheduler] DEBUG o.e.jetty.client.HttpConnection - Released HttpChannelOverHTTP2@76497e77(exchange=null)[send=HttpSenderOverHTTP2@37cd9630(req=QUEUED,snd=COMPLETED,failure=null),recv=HttpReceiverOverHTTP2@3e32101a(rsp=FAILURE,failure=java.util.concurrent.TimeoutException: Total timeout 6000 ms elapsed)]
22:48:34.269 [HttpClient@68be2bc2-scheduler] DEBUG o.eclipse.jetty.client.HttpReceiver - Request/Response failed: Result[HttpRequest[POST /big-body HTTP/2.0]@3b088d51 > HttpResponse[HTTP/2.0 200 null]@2d95c30d] java.util.concurrent.TimeoutException: Total timeout 6000 ms elapsed
22:48:34.269 [HttpClient@68be2bc2-scheduler] DEBUG o.eclipse.jetty.client.HttpExchange - Aborted (true) while active HttpExchange@6ce62fb9 req=TERMINATED/null@null res=TERMINATED/java.util.concurrent.TimeoutException: Total timeout 6000 ms elapsed@3ff6c3b1: {}
22:48:34.269 [HttpClient@68be2bc2-scheduler] DEBUG o.e.jetty.client.HttpDestination - org.eclipse.jetty.client.HttpDestination$TimeoutTask@53f65459 timeout expired
22:48:34.270 [main] DEBUG o.e.jetty.client.HttpDestination - Closed HttpDestination[http://127.0.0.1:7004]@7f13d6e,queue=0,pool=MultiplexConnectionPool@51cdd8a[c=1/64,b=0,m=0,i=1]
22:48:34.271 [main] DEBUG o.e.j.client.AbstractConnectionPool - Connection removed HttpConnectionOverHTTP2@7cb68078(closed=true)[HTTP2ClientSession@20fef01c{l:/127.0.0.1:55074 <-> r:/127.0.0.1:7004,sendWindow=65535,recvWindow=16772681,streams=0,NOT_CLOSED,null}] - pooled: -1
Exception in thread "main" java.util.concurrent.TimeoutException: Total timeout 6000 ms elapsed
	at org.eclipse.jetty.client.TimeoutCompleteListener.onTimeoutExpired(TimeoutCompleteListener.java:51)
	at org.eclipse.jetty.io.CyclicTimeout$Wakeup.run(CyclicTimeout.java:282)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

[qingdaoheze]

I can get the correct response by following curl command.
curl -v --http2 --compressed -X POST "http://127.0.0.1:7004/big-body?size=1000" --compressed -H "accept: /" -H "Accept-Encoding: gzip" -H "x-gray-env: normal"

• Trying 127.0.0.1...
• TCP_NODELAY set
• Connected to 127.0.0.1 (127.0.0.1) port 7004 (#0)

POST /big-body?size=1000 HTTP/1.1
Host: 127.0.0.1:7004
User-Agent: curl/7.64.1
Connection: Upgrade, HTTP2-Settings
Upgrade: h2c
HTTP2-Settings: AAMAAABkAARAAAAAAAIAAAAA
accept: /
Accept-Encoding: gzip
x-gray-env: normal

< HTTP/1.1 101
< Connection: Upgrade
< Upgrade: h2c
< Date: Tue, 06 Apr 2021 14:53:32 GMT

• Received 101
• Using HTTP2, server supports multi-use
• Connection state changed (HTTP/2 confirmed)
• Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
• Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
  < HTTP/2 200
  < content-type: application/json;charset=UTF-8
  < content-length: 19781
  < date: Tue, 06 Apr 2021 14:53:32 GMT
  < vary: accept-encoding
  < content-encoding: gzip
  <
  {"key297":"value297","key296":"value296","key299":"value299","key298":"value298","key293":"value293","key292":"value292","key295":"value295","key294":"value294","key291":"value291","key290":"value290","key914":"value914","key913":"value913","key916":"value916","key915":"value915","key910":"value910","key912":"value912","key911":"value911","key907":"value907","key906":"value906","key909":"value909","key908":"value908","key925":"value925","key924":"value924","key927":"value927","key926":"value926","key921":"value921","key920":"value920","key923":"value923","key922":"value922","key918":"value918","key917":"value917","key919":"value919","key903":"value903","key902":"value902","key905":"value905","key904":"value904","key901":"value901","key900":"value900","key716":"value716","key958":"value958","key715":"value715","key957":"value957","key718":"value718","key717":"value717","key959":"value959","key712":"value712","key954":"value954","key711":"value711","key953":"value953","key714":"value714","key956":"value956","key713":"value713","key955":"value955","key950":"value950","key710":"value710","key952":"value952","key951":"value951","key709":"value709","key708":"value708","key727":"value727","key969":"value969","key726":"value726","key968":"value968","key729":"value729","key728":"value728","key723":"value723","key965":"value965","key722":"value722","key964":"value964","key725":"value725","key967":"value967","key724":"value724","key966":"value966","key961":"value961","key960":"value960","key721":"value721","key963":"value963","key720":"value720","key962":"value962","key719":"value719","key936":"value936","key935":"value935","key938":"value938","key937":"value937","key932":"value932","key931":"value931","key934":"value934","key933":"value933","key930":"value930","key929":"value929","key928":"value928","key705":"value705","key947":"value947","key704":"value704","key946":"value946","key707":"value707","key949":"value949","key706":"value706","key948":"value948","key701":"value701","key943":"value943","key700":"value700","key942":"value942","key703":"value703","key945":"value945","key702":"value702","key944":"value944","key941":"value941","key940":"value940","key939":"value939","key518":"value518","key517":"value517","key759":"value759","key519":"value519","key514":"value514","key756":"value756","key998":"value998","key513":"value513","key755":"value755","key997":"value997","key516":"value516","key758":"value758","key515":"value515","key757":"value757","key999":"value999","key510":"value510","key752":"value752","key994":"value994","key751":"value751","key993":"value993","key512":"value512","key754":"value754","key996":"value996","key511":"value511","key753":"value753","key995":"value995","key990":"value990","key750":"value750","key992":"value992","key991":"value991","key529":"value529","key528":"value528","key525":"value525","key767":"value767","key524":"value524","key766":"value766","key527":"value527","key769":"value769","key526":"value526","key768":"value768","key521":"value521","key763":"value763","key520":"value520","key762":"value762","key523":"value523","key765":"value765","key522":"value522","key764":"value764","key761":"value761","key760":"value760","key738":"value738","key737":"value737","key979":"value979","key739":"value739","key734":"value734","key976":"value976","key733":"value733","key975":"value975","key736":"value736","key978":"value978","key735":"value735","key977":"value977","key730":"value730","key972":"value972","key971":"value971","key732":"value732","key974":"value974","key731":"value731","key973":"value973","key970":"value970","key507":"value507","key749":"value749","key506":"value506","key748":"value748","key509":"value509","key508":"value508","key503":"value503","key745":"value745","key987":"value987","key502":"value502","key744":"value744","key986":"value986","key505":"value505","key747":"value747","key989":"value989","key504":"value504","key746":"value746","key988":"value988","key741":"value741","key983":"value983","key740":"value740","key982":"value982","key501":"value501","key743":"value743","key985":"value985","key500":"value500","key742":"value742","key984":"value984","key981":"value981","key980":"value980","key319":"value319","key316":"value316","key558":"value558","key315":"value315","key557":"value557","key799":"value799","key318":"value318","key317":"value317","key559":"value559","key312":"value312","key554":"value554","key796":"value796","key311":"value311","key553":"value553","key795":"value795","key314":"value314","key556":"value556","key798":"value798","key313":"value313","key555":"value555","key797":"value797","key550":"value550","key792":"value792","key791":"value791","key310":"value310","key552":"value552","key794":"value794","key551":"value551","key793":"value793","key790":"value790","key327":"value327","key569":"value569","key326":"value326","key568":"value568","key329":"value329","key328":"value328","key323":"value323","key565":"value565","key322":"value322","key564":"value564","key325":"value325","key567":"value567","key324":"value324","key566":"value566","key561":"value561","key560":"value560","key321":"value321","key563":"value563","key320":"value320","key562":"value562","key539":"value539","key536":"value536","key778":"value778","key535":"value535","key777":"value777","key538":"value538","key537":"value537","key779":"value779","key532":"value532","key774":"value774","key531":"value531","key773":"value773","key534":"value534","key776":"value776","key533":"value533","key775":"value775","key770":"value770","key530":"value530","key772":"value772","key771":"value771","key309":"value309","key308":"value308","key305":"value305","key547":"value547","key789":"value789","key304":"value304","key546":"value546","key788":"value788","key307":"value307","key549":"value549","key306":"value306","key548":"value548","key301":"value301","key543":"value543","key785":"value785","key300":"value300","key542":"value542","key784":"value784","key303":"value303","key545":"value545","key787":"value787","key302":"value302","key544":"value544","key786":"value786","key781":"value781","key780":"value780","key541":"value541","key783":"value783","key540":"value540","key782":"value782","key118":"value118","key117":"value117","key359":"value359","key119":"value119","key114":"value114","key356":"value356","key598":"value598","key113":"value113","key355":"value355","key597":"value597","key116":"value116","key358":"value358","key115":"value115","key357":"value357","key599":"value599","key110":"value110","key352":"value352","key594":"value594","key351":"value351","key593":"value593","key112":"value112","key354":"value354","key596":"value596","key111":"value111","key353":"value353","key595":"value595","key590":"value590","key350":"value350","key592":"value592","key591":"value591","key129":"value129","key128":"value128","key125":"value125","key367":"value367","key124":"value124","key366":"value366","key127":"value127","key369":"value369","key126":"value126","key368":"value368","key121":"value121","key363":"value363","key120":"value120","key362":"value362","key123":"value123","key365":"value365","key122":"value122","key364":"value364","key361":"value361","key360":"value360","key338":"value338","key337":"value337","key579":"value579","key339":"value339","key334":"value334","key576":"value576","key333":"value333","key575":"value575","key336":"value336","key578":"value578","key335":"value335","key577":"value577","key330":"value330","key572":"value572","key571":"value571","key332":"value332","key574":"value574","key331":"value331","key573":"value573","key570":"value570","key107":"value107","key349":"value349","key106":"value106","key348":"value348","key109":"value109","key108":"value108","key103":"value103","key345":"value345","key587":"value587","key102":"value102","key344":"value344","key586":"value586","key105":"value105","key347":"value347","key589":"value589","key104":"value104","key346":"value346","key588":"value588","key341":"value341","key583":"value583","key340":"value340","key582":"value582","key101":"value101","key343":"value343","key585":"value585","key100":"value100","key342":"value342","key584":"value584","key581":"value581","key580":"value580","key158":"value158","key157":"value157","key399":"value399","key159":"value159","key154":"value154","key396":"value396","key153":"value153","key395":"value395","key156":"value156","key398":"value398","key155":"value155","key397":"value397","key150":"value150","key392":"value392","key391":"value391","key152":"value152","key394":"value394","key151":"value151","key393":"value393","key390":"value390","key169":"value169","key168":"value168","key165":"value165","key164":"value164","key167":"value167","key166":"value166","key161":"value161","key160":"value160","key163":"value163","key162":"value162","key139":"value139","key136":"value136","key378":"value378","key135":"value135","key377":"value377","key138":"value138","key137":"value137","key379":"value379","key132":"value132","key374":"value374","key131":"value131","key373":"value373","key134":"value134","key376":"value376","key133":"value133","key375":"value375","key370":"value370","key130":"value130","key372":"value372","key371":"value371","key147":"value147","key389":"value389","key146":"value146","key388":"value388","key149":"value149","key148":"value148","key143":"value143","key385":"value385","key142":"value142","key384":"value384","key145":"value145","key387":"value387","key144":"value144","key386":"value386","key381":"value381","key380":"value380","key141":"value141","key383":"value383","key140":"value140","key382":"value382","key198":"value198","key197":"value197","key199":"value199","key194":"value194","key193":"value193","key196":"value196","key195":"value195","key190":"value190","key192":"value192","key191":"value191","key179":"value179","key176":"value176","key175":"value175","key178":"value178","key177":"value177","key172":"value172","key171":"value171","key174":"value174","key173":"value173","key170":"value170","key187":"value187","key186":"value186","key189":"value189","key188":"value188","key183":"value183","key182":"value182","key185":"value185","key184":"value184","key181":"value181","key180":"value180","key804":"value804","key803":"value803","key806":"value806","key805":"value805","key800":"value800","key802":"value802","key801":"value801","key1":"value1","key837":"value837","key2":"value2","key836":"value836","key839":"value839","key0":"value0","key838":"value838","key5":"value5","key833":"value833","key6":"value6","key832":"value832","key3":"value3","key835":"value835","key4":"value4","key834":"value834","key9":"value9","key7":"value7","key831":"value831","key8":"value8","key830":"value830","key829":"value829","key606":"value606","key848":"value848","key605":"value605","key847":"value847","key608":"value608","key607":"value607","key849":"value849","key602":"value602","key844":"value844","key601":"value601","key843":"value843","key604":"value604","key846":"value846","key603":"value603","key845":"value845","key840":"value840","key600":"value600","key842":"value842","key841":"value841","key815":"value815","key814":"value814","key817":"value817","key816":"value816","key811":"value811","key810":"value810","key813":"value813","key812":"value812","key808":"value808","key807":"value807","key809":"value809","key826":"value826","key825":"value825","key828":"value828","key827":"value827","key822":"value822","key821":"value821","key824":"value824","key823":"value823","key820":"value820","key819":"value819","key818":"value818","key639":"value639","key638":"value638","key635":"value635","key877":"value877","key634":"value634","key876":"value876","key637":"value637","key879":"value879","key636":"value636","key878":"value878","key631":"value631","key873":"value873","key630":"value630","key872":"value872","key633":"value633","key875":"value875","key632":"value632","key874":"value874","key871":"value871","key870":"value870","key408":"value408","key407":"value407","key649":"value649","key409":"value409","key404":"value404","key646":"value646","key888":"value888","key403":"value403","key645":"value645","key887":"value887","key406":"value406","key648":"value648","key405":"value405","key647":"value647","key889":"value889","key400":"value400","key642":"value642","key884":"value884","key641":"value641","key883":"value883","key402":"value402","key644":"value644","key886":"value886","key401":"value401","key643":"value643","key885":"value885","key880":"value880","key640":"value640","key882":"value882","key881":"value881","key617":"value617","key859":"value859","key616":"value616","key858":"value858","key619":"value619","key618":"value618","key613":"value613","key855":"value855","key612":"value612","key854":"value854","key615":"value615","key857":"value857","key614":"value614","key856":"value856","key851":"value851","key850":"value850","key611":"value611","key853":"value853","key610":"value610","key852":"value852","key609":"value609","key628":"value628","key627":"value627","key869":"value869","key629":"value629","key624":"value624","key866":"value866","key623":"value623","key865":"value865","key626":"value626","key868":"value868","key625":"value625","key867":"value867","key620":"value620","key862":"value862","key861":"value861","key622":"value622","key864":"value864","key621":"value621","key863":"value863","key860":"value860","key437":"value437","key679":"value679","key436":"value436","key678":"value678","key439":"value439","key438":"value438","key433":"value433","key675":"value675","key432":"value432","key674":"value674","key435":"value435","key677":"value677","key434":"value434","key676":"value676","key671":"value671","key670":"value670","key431":"value431","key673":"value673","key430":"value430","key672":"value672","key209":"value209","key206":"value206","key448":"value448","key205":"value205","key447":"value447","key689":"value689","key208":"value208","key207":"value207","key449":"value449","key202":"value202","key444":"value444","key686":"value686","key201":"value201","key443":"value443","key685":"value685","key204":"value204","key446":"value446","key688":"value688","key203":"value203","key445":"value445","key687":"value687","key11":"value11","key440":"value440","key682":"value682","key10":"value10","key681":"value681","key200":"value200","key442":"value442","key684":"value684","key441":"value441","key683":"value683","key680":"value680","key419":"value419","key418":"value418","key415":"value415","key657":"value657","key899":"value899","key414":"value414","key656":"value656","key898":"value898","key417":"value417","key659":"value659","key416":"value416","key658":"value658","key411":"value411","key653":"value653","key895":"value895","key410":"value410","key652":"value652","key894":"value894","key413":"value413","key655":"value655","key897":"value897","key412":"value412","key654":"value654","key896":"value896","key22":"value22","key891":"value891","key21":"value21","key890":"value890","key20":"value20","key651":"value651","key893":"value893","key650":"value650","key892":"value892","key15":"value15","key14":"value14","key13":"value13","key12":"value12","key19":"value19","key18":"value18","key17":"value17","key16":"value16","key429":"value429","key426":"value426","key668":"value668","key425":"value425","key667":"value667","key428":"value428","key427":"value427","key669":"value669","key422":"value422","key664":"value664","key421":"value421","key663":"value663","key424":"value424","key666":"value666","key423":"value423","key665":"value665","key33":"value33","key660":"value660","key32":"value32","key31":"value31","key420":"value420","key662":"value662","key30":"value30","key661":"value661","key26":"value26","key25":"value25","key24":"value24","key23":"value23","key29":"value29","key28":"value28","key27":"value27","key239":"value239","key238":"value238","key40":"value40","key235":"value235","key477":"value477","key234":"value234","key476":"value476","key237":"value237","key479":"value479","key236":"value236","key478":"value478","key44":"value44","key231":"value231","key473":"value473","key43":"value43","key230":"value230","key472":"value472","key42":"value42","key233":"value233","key475":"value475","key41":"value41","key232":"value232","key474":"value474","key37":"value37","key36":"value36","key35":"value35","key471":"value471","key34":"value34","key470":"value470","key39":"value39","key38":"va* transfer closed with 15246 bytes remaining to read
• stopped the pause stream!
• Connection #0 to host 127.0.0.1 left intact
  curl: (18) transfer closed with 15246 bytes remaining to read
  lue38","key249":"value249","key51":"value51","key246":"value246","key488":"value488","key50":"value50","key245":"value245","key487":"value487","key248":"value248","key247":"value247","key489":"value489","key55":"value55","key242":"value242","key484":"value484","key54":"value54","key241":"value241","key483":"value483","key53":"value53","key244":"value244","key486":"value486","key52":"value52","key243":"value243","key485":"value485","key48":"value48","key480":"value480","key47":"value47","key46":"value46","key240":"value240","key482":"value482","key45":"value45","key481":"value481","key49":"value49","key217":"value217","key459":"value459","key216":"value216","key458":"value458","key219":"value219","key218":"value218","key62":"value62","key213":"value213","key455":"value455","key697":"value697","key61":"value61","key212":"value212","key454":"value454","key696":"value696","key60":"value60","key215":"value215","key457":"value457","key699":"value699","key214":"value214","key456":"value456","key698":"value698","key66":"value66","key451":"value451","key693":"value693","key65":"value65","key450":"value450","key692":"value692","key64":"value64","key211":"value211","key453":"value453","key695":"value695","key63":"value63","key210":"value210","key452":"value452","key694":"value694","key59":"value59","key58":"value58","key57":"value57","key691":"value691","key56":"value56","key690":"value690","key228":"value228","key227":"value227","key469":"value469","key229":"value229","key73":"value73","key224":"value224","key466":"value466","key72":"value72","key223":"value223","key465":"value465","key71":"value71","key226":"value226","key468":"value468","key70":"value70","key225":"value225","key467":"value467","key77":"value77","key220":"value220","key462":"value462","key76":"value76","key461":"value461","key75":"value75","key222":"value222","key464":"value464","key74":"value74","key221":"value221","key463":"value463","key69":"value69","key68":"value68","key460":"value460","key67":"value67","key80":"value80","key84":"value84","key279":"value279","key83":"value83","key278":"value278","key82":"value82","key81":"value81","key88":"value88","key275":"value275","key87":"value87","key274":"value274","key86":"value86","key277":"value277","key85":"value85","key276":"value276","key271":"value271","key270":"value270","key79":"value79","key273":"value273","key78":"value78","key272":"value272","key91":"value91","key90":"value90","key95":"value95","key94":"value94","key289":"value289","key93":"value93","key92":"value92","key99":"value99","key286":"value286","key98":"value98","key285":"value285","key97":"value97","key288":"value288","key96":"value96","key287":"value287","key282":"value282","key281":"value281","key284":"value284","key89":"value89","key283":"value283","key280":"value280","key257":"value257","key499":"value499","key256":"value256","key498":"value498","key259":"value259","key258":"value258","key253":"value253","key495":"value495","key252":"value252","key494":"value494","key255":"value255","key497":"value497","key254":"value254","key496":"value496","key491":"value491","key490":"value490","key251":"value251","key493":"value493","key250":"value250","key492":"value492","key268":"value268","key267":"value267","key269":"value269","key264":"value264","key263":"value263","key266":"value266","key265":"value265","key260":"value260","key262":"value262","key261":"value261"}* Closing connection 0

[linking12]

same issue, any workaround?

[sbordet]

I cannot reproduce this issue with the latest Jetty, everything works fine for me.

Can you please enable DEBUG logging for org.eclipse.jetty (not just org.eclipse.jetty.client), save the logs into a file and attach the log file to this issue?

Your current logs show that the client sent a reset to the server after receiving the first 2 chunks.
We need the full DEBUG logs to understand what happened at the HTTP/2 level.

[qingdaoheze]

https://github.com/qingdaoheze/jetty-gzip-decode

This is my demo projects, you can reproduce the case as below.
1.Start tomcat server
tomcat-server-gzip/src/main/java/org/sample/http2/tomcat/TomcatHttp2Main.java
2.Call the service with Jetty 9 client, and the error will occur.
jetty9-client-sample/src/main/java/org/sample/jetty/client/http2/HttpClientWithHttp2Transport.java
3.Call the service with ok-http, it is normal.
okhttp-client-sample/src/main/java/sample/okhttp/h2c/Http2ClearTextSample.java

[sbordet]

@qingdaoheze I took your repository, fixed the POMs, and run your project.
Works perfectly fine for me:

2021-04-07 16:23:49.308:INFO::main: Logging initialized @119ms to org.eclipse.jetty.util.log.StdErrLog
2021-04-07 16:23:49.500:INFO:osjch.HttpClientWithHttp2Transport:main: status:200
2021-04-07 16:23:49.501:INFO:osjch.HttpClientWithHttp2Transport:main: version: HTTP/2.0
2021-04-07 16:23:49.501:INFO:osjch.HttpClientWithHttp2Transport:main: response:name=string0,age=0,name=string1,age=1,name=string2,age=2,name=string3,age=3,name=string4,age=4,name=string5,age=5,name=string6,age=6,name=string7,age=7,name=string8,age=8,name=string9,age=9,name=string10,age=10,name=string11,age=11,name=string12,age=12,name=string13,age=13,name=string14,age=14,name=string15,age=15,name=string16,age=16,name=string17,age=17,name=string18,age=18,name=string19,age=19,name=string20,age=20,name=string21,age=21,name=string22,age=22,name=string23,age=23,name=string24,age=24,name=string25,age=25,name=string26,age=26,name=string27,age=27,name=string28,age=28,name=string29,age=29,name=string30,age=30,name=string31,age=31,name=string32,age=32,name=string33,age=33,name=string34,age=34,name=string35,age=35,name=string36,age=36,name=string37,age=37,name=string38,age=38,name=string39,age=39,name=string40,age=40,name=string41,age=41,name=string42,age=42,name=string43,age=43,name=string44,age=44,name=string45,age=45,name=string46,age=46,name=string47,age=47,name=string48,age=48,name=string49,age=49,name=string50,age=50,name=string51,age=51,name=string52,age=52,name=string53,age=53,name=string54,age=54,name=string55,age=55,name=string56,age=56,name=string57,age=57,name=string58,age=58,name=string59,age=59,name=string60,age=60,name=string61,age=61,name=string62,age=62,name=string63,age=63,name=string64,age=64,name=string65,age=65,name=string66,age=66,name=string67,age=67,name=string68,age=68,name=string69,age=69,name=string70,age=70,name=string71,age=71,name=string72,age=72,name=string73,age=73,name=string74,age=74,name=string75,age=75,name=string76,age=76,name=string77,age=77,name=string78,age=78,name=string79,age=79,name=string80,age=80,name=string81,age=81,name=string82,age=82,name=string83,age=83,name=string84,age=84,name=string85,age=85,name=string86,age=86,name=string87,age=87,name=string88,age=88,name=string89,age=89,name=string90,age=90,name=string91,age=91,name=string92,age=92,name=string93,age=93,name=string94,age=94,name=string95,age=95,name=string96,age=96,name=string97,age=97,name=string98,age=98,name=string99,age=99,name=string100,age=100,name=string101,age=101,name=string102,age=102,name=string103,age=103,name=string104,age=104,name=string105,age=105,name=string106,age=106,name=string107,age=107,name=string108,age=108,name=string109,age=109,name=string110,age=110,name=string111,age=111,name=string112,age=112,name=string113,age=113,name=string114,age=114,name=string115,age=115,name=string116,age=116,name=string117,age=117,name=string118,age=118,name=string119,age=119,name=string120,age=120,name=string121,age=121,name=string122,age=122,name=string123,age=123,name=string124,age=124,name=string125,age=125,name=string126,age=126,name=string127,age=127,name=string128,age=128,name=string129,age=129,name=string130,age=130,name=string131,age=131,name=string132,age=132,name=string133,age=133,name=string134,age=134,name=string135,age=135,name=string136,age=136,name=string137,age=137,name=string138,age=138,name=string139,age=139,name=string140,age=140,name=string141,age=141,name=string142,age=142,name=string143,age=143,name=string144,age=144,name=string145,age=145,name=string146,age=146,name=string147,age=147,name=string148,age=148,name=string149,age=149,name=string150,age=150,name=string151,age=151,name=string152,age=152,name=string153,age=153,name=string154,age=154,name=string155,age=155,name=string156,age=156,name=string157,age=157,name=string158,age=158,name=string159,age=159,name=string160,age=160,name=string161,age=161,name=string162,age=162,name=string163,age=163,name=string164,age=164,name=string165,age=165,name=string166,age=166,name=string167,age=167,name=string168,age=168,name=string169,age=169,name=string170,age=170,name=string171,age=171,name=string172,age=172,name=string173,age=173,name=string174,age=174,name=string175,age=175,name=string176,age=176,name=string177,age=177,name=string178,age=178,name=string179,age=179,name=string180,age=180,name=string181,age=181,name=string182,age=182,name=string183,age=183,name=string184,age=184,name=string185,age=185,name=string186,age=186,name=string187,age=187,name=string188,age=188,name=string189,age=189,name=string190,age=190,name=string191,age=191,name=string192,age=192,name=string193,age=193,name=string194,age=194,name=string195,age=195,name=string196,age=196,name=string197,age=197,name=string198,age=198,name=string199,age=199,
2021-04-07 16:23:49.501:INFO:osjch.HttpClientWithHttp2Transport:main: contentEncoding:gzip
HTTP/2.0
name=string0,age=0,name=string1,age=1,name=string2,age=2,name=string3,age=3,name=string4,age=4,name=string5,age=5,name=string6,age=6,name=string7,age=7,name=string8,age=8,name=string9,age=9,name=string10,age=10,name=string11,age=11,name=string12,age=12,name=string13,age=13,name=string14,age=14,name=string15,age=15,name=string16,age=16,name=string17,age=17,name=string18,age=18,name=string19,age=19,name=string20,age=20,name=string21,age=21,name=string22,age=22,name=string23,age=23,name=string24,age=24,name=string25,age=25,name=string26,age=26,name=string27,age=27,name=string28,age=28,name=string29,age=29,name=string30,age=30,name=string31,age=31,name=string32,age=32,name=string33,age=33,name=string34,age=34,name=string35,age=35,name=string36,age=36,name=string37,age=37,name=string38,age=38,name=string39,age=39,name=string40,age=40,name=string41,age=41,name=string42,age=42,name=string43,age=43,name=string44,age=44,name=string45,age=45,name=string46,age=46,name=string47,age=47,name=string48,age=48,name=string49,age=49,name=string50,age=50,name=string51,age=51,name=string52,age=52,name=string53,age=53,name=string54,age=54,name=string55,age=55,name=string56,age=56,name=string57,age=57,name=string58,age=58,name=string59,age=59,name=string60,age=60,name=string61,age=61,name=string62,age=62,name=string63,age=63,name=string64,age=64,name=string65,age=65,name=string66,age=66,name=string67,age=67,name=string68,age=68,name=string69,age=69,name=string70,age=70,name=string71,age=71,name=string72,age=72,name=string73,age=73,name=string74,age=74,name=string75,age=75,name=string76,age=76,name=string77,age=77,name=string78,age=78,name=string79,age=79,name=string80,age=80,name=string81,age=81,name=string82,age=82,name=string83,age=83,name=string84,age=84,name=string85,age=85,name=string86,age=86,name=string87,age=87,name=string88,age=88,name=string89,age=89,name=string90,age=90,name=string91,age=91,name=string92,age=92,name=string93,age=93,name=string94,age=94,name=string95,age=95,name=string96,age=96,name=string97,age=97,name=string98,age=98,name=string99,age=99,name=string100,age=100,name=string101,age=101,name=string102,age=102,name=string103,age=103,name=string104,age=104,name=string105,age=105,name=string106,age=106,name=string107,age=107,name=string108,age=108,name=string109,age=109,name=string110,age=110,name=string111,age=111,name=string112,age=112,name=string113,age=113,name=string114,age=114,name=string115,age=115,name=string116,age=116,name=string117,age=117,name=string118,age=118,name=string119,age=119,name=string120,age=120,name=string121,age=121,name=string122,age=122,name=string123,age=123,name=string124,age=124,name=string125,age=125,name=string126,age=126,name=string127,age=127,name=string128,age=128,name=string129,age=129,name=string130,age=130,name=string131,age=131,name=string132,age=132,name=string133,age=133,name=string134,age=134,name=string135,age=135,name=string136,age=136,name=string137,age=137,name=string138,age=138,name=string139,age=139,name=string140,age=140,name=string141,age=141,name=string142,age=142,name=string143,age=143,name=string144,age=144,name=string145,age=145,name=string146,age=146,name=string147,age=147,name=string148,age=148,name=string149,age=149,name=string150,age=150,name=string151,age=151,name=string152,age=152,name=string153,age=153,name=string154,age=154,name=string155,age=155,name=string156,age=156,name=string157,age=157,name=string158,age=158,name=string159,age=159,name=string160,age=160,name=string161,age=161,name=string162,age=162,name=string163,age=163,name=string164,age=164,name=string165,age=165,name=string166,age=166,name=string167,age=167,name=string168,age=168,name=string169,age=169,name=string170,age=170,name=string171,age=171,name=string172,age=172,name=string173,age=173,name=string174,age=174,name=string175,age=175,name=string176,age=176,name=string177,age=177,name=string178,age=178,name=string179,age=179,name=string180,age=180,name=string181,age=181,name=string182,age=182,name=string183,age=183,name=string184,age=184,name=string185,age=185,name=string186,age=186,name=string187,age=187,name=string188,age=188,name=string189,age=189,name=string190,age=190,name=string191,age=191,name=string192,age=192,name=string193,age=193,name=string194,age=194,name=string195,age=195,name=string196,age=196,name=string197,age=197,name=string198,age=198,name=string199,age=199,

[qingdaoheze]

What did you change? @sbordet

[sbordet]

Made the root POM a multi-module, added the correct <parent> to child modules, specified the spring-boot version in the dependencies, replaced LogBack with Jetty logging, nothing major.

No changes to the code, everything works fine.

[qingdaoheze]

What JDK version do you use?

[qingdaoheze]

I added the JDK version in the pom. Please try again, And I add you to the collaborator of my repo, you can modify it directly. @sbordet

[linking12]

I think tomcat 9.0.21 is not suit for jetty http2， when upgrade to 9.0.44 work fine

[qingdaoheze]

I upgraded the tomcat version, it works fine. Thank you very much!