SSLHandshakeException in Jetty client when reading long files over SSL

[nniv-r7]

Jetty version
Jetty client 9.4.39.v20210325

Java version
1.8.0_281

OS type/version
Windows 10

Description
While reading long files over SSL (specifically from AWS S3 resource), short files (a few Kbytes) are read OK, but reading a long file terminates with thrown exception "Encrypted buffer max length exceeded", with the stacktrace below. Since it doesn't happen with 9.4.38.v20210224 (or earlier) I assume it's a regression issue.

[{:type javax.net.ssl.SSLHandshakeException
:message "Encrypted buffer max length exceeded"
:at [org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint fill "SslConnection.java" 735]}]
:trace
[[org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint fill "SslConnection.java" 735]
[org.eclipse.jetty.client.http.HttpReceiverOverHTTP process "HttpReceiverOverHTTP.java" 164]
[org.eclipse.jetty.client.http.HttpReceiverOverHTTP receive "HttpReceiverOverHTTP.java" 79]
[org.eclipse.jetty.client.http.HttpChannelOverHTTP receive "HttpChannelOverHTTP.java" 131]
[org.eclipse.jetty.client.http.HttpConnectionOverHTTP onFillable "HttpConnectionOverHTTP.java" 169]
[org.eclipse.jetty.io.AbstractConnection$ReadCallback succeeded "AbstractConnection.java" 311]
[org.eclipse.jetty.io.FillInterest fillable "FillInterest.java" 105]
[org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint onFillable "SslConnection.java" 540]
[org.eclipse.jetty.io.ssl.SslConnection onFillable "SslConnection.java" 395]
[org.eclipse.jetty.io.ssl.SslConnection$2 succeeded "SslConnection.java" 161]
[org.eclipse.jetty.io.FillInterest fillable "FillInterest.java" 105]
[org.eclipse.jetty.io.ChannelEndPoint$1 run "ChannelEndPoint.java" 104]
[org.eclipse.jetty.util.thread.strategy.EatWhatYouKill runTask "EatWhatYouKill.java" 336]
[org.eclipse.jetty.util.thread.strategy.EatWhatYouKill doProduce "EatWhatYouKill.java" 313]
[org.eclipse.jetty.util.thread.strategy.EatWhatYouKill tryProduce "EatWhatYouKill.java" 171]
[org.eclipse.jetty.util.thread.strategy.EatWhatYouKill run "EatWhatYouKill.java" 129]
[org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread run "ReservedThreadExecutor.java" 383]
[org.eclipse.jetty.util.thread.QueuedThreadPool runJob "QueuedThreadPool.java" 882]
[org.eclipse.jetty.util.thread.QueuedThreadPool$Runner run "QueuedThreadPool.java" 1036]
[java.lang.Thread run "Thread.java" 748]]

[joakime]

The message "Encrypted buffer max length exceeded" was introduced as part of PR #6073

[joakime]

Can you run your client with the JVM System Property -Djsse.SSLEngine.acceptLargeFragments=true and see if your AWS S3 tests are happy again?

[nniv-r7]

I tried with the system property you suggested - it doesn't fix the issue, the exception is still thrown as described

[joakime]

I'd like to try to replicate this.
What size file are you downloading?
Can you run with DEBUG logging on? (org.eclipse.jetty.io.LEVEL=DEBUG)

[nniv-r7]

Attached the log (with some values redacted), I hope this helps. I believe there are 1 (or 2) successful reads of short file(s) and then a failure on reading a long file.
jetty-log.txt

[lorban]

@nniv-r7 that seems to be a regression in 9.4.39 that was introduced as part of PR #6073 as @joakime suggested. We're now busy investigating the best way to test and fix it.

The change that was applied to Jetty 10.0.2 (and 11.0.2) is slightly different and it seems that these versions wouldn't have this problem.

Could it be a possibility for you to try out Jetty 10.0.2 and see if you can reproduce the problem with it?

[nniv-r7]

I'll need to check that I can switch this specific usecase from Java 8 to 11 first for that - I'll try and will update if/when I can get it to run.

[anilkumarmyla]

We see this issue too, let me know if I can help in any way

[sbordet]

@anilkumarmyla can you try Jetty 10.0.2 and see if the problem is solved?

[tbutter]

We see the same problem with server code getting a large POST from AWS Cloudfront. Works fine in 6.4.38 / fails most of the time in 6.4.39.

[gregw]

If 9.4.38 is working, but without going 100% CPU, I expect that this is a compaction problem... ie there is space in the buffer, but only if we compact it again.
So it would be very interesting to try 10.0.2, which has an improved approach to compaction (#6082) and will try a compact to make more space before throwing this exception.

Perhaps we should prepare a branch with #6082 cherry-picked back to 9?

[gregw]

Looking at 9.4.39 again, I think this may be related to this fix for an android bug. The fix fakes an UNDERFLOW state if no bytes were produced or consumed (which might happen if the SslEngine just wanted task run) and worked because the previous handling of UNDERFLOW was to do a go-around, which would compact and try again. So with 9.4.39, a buffer that is completely filled, but that could be compressed will cause the problem if this fake UNDERFLOW is hit.

I definitely believe #6082 will resolve this.

[sbordet]

Fixed by #6142.