0-byte gzip response causes java.io.EOFException

[igokoro]

I'm using Retrofit (1.9.0) with OhHttp/okio (2.3.0/1.3.0) to implement communication with OneDrive REST API. While almost everything works amazingly well, there is a problem with java.io.EOFException on action.copy.

OneDrive server returns

HTTP 202 Accepted
Content-Length: 0
Content-Encoding: gzip

Under the hood OkHttp includes Accept-Encoding: gzip into request headers, so server legitimately returns a 0-byte gzip-like response.

One the response side, com.squareup.okhttp.internal.http.HttpEngine unzips response into GzipSource in method unzip(Response). The problem is that GzipSource tries to consume 10-byte gzip header from 0-length stream, so the java.io.EOFException is thrown because of seemingly missing 10 bytes.

My guess is that the problem is in the HttpEngine.unzip(Response) method: it checks if response is actually compressed and if the response body is not null. But it fails to check content length to know if there is something is the stream.

The problem is even worse because the original Content-Length and Content-Encoding headers are stripped off and on the client side there is no way to know if it's safe read from the response stream.

Here is the original stack trace from my application:

retrofit.RetrofitError
            at retrofit.RestAdapter$RestHandler.invokeRequest(RestAdapter.java:395)
            at retrofit.RestAdapter$RestHandler.invoke(RestAdapter.java:240)
            at java.lang.reflect.Proxy.invoke(Proxy.java:397)
            at $Proxy13.copy(Unknown Source)
            at com.ncryptedcloud.cloudprovider.onedrive.OneDriveCloudProvider.copy(OneDriveCloudProvider.java:731)
            at com.ncryptedcloud.operation.cloudprovider.CopyFileOperation.execute(CopyFileOperation.java:72)
            at com.foxykeep.datadroid.service.RequestService.onHandleIntent(RequestService.java:145)
            at com.foxykeep.datadroid.service.MultiThreadedIntentService$IntentRunnable.run(MultiThreadedIntentService.java:170)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:422)
            at java.util.concurrent.FutureTask.run(FutureTask.java:237)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
            at java.lang.Thread.run(Thread.java:818)
     Caused by: java.io.EOFException
            at okio.RealBufferedSource.require(RealBufferedSource.java:64)
            at okio.GzipSource.consumeHeader(GzipSource.java:114)
            at okio.GzipSource.read(GzipSource.java:73)
            at okio.RealBufferedSource$1.read(RealBufferedSource.java:338)
            at java.io.InputStream.read(InputStream.java:162)
            at retrofit.Utils.streamToBytes(Utils.java:43)
            at retrofit.Utils.readBodyToBytesIfNecessary(Utils.java:81)
            at retrofit.RestAdapter$RestHandler.invokeRequest(RestAdapter.java:348)
            at retrofit.RestAdapter$RestHandler.invoke(RestAdapter.java:240)
            at java.lang.reflect.Proxy.invoke(Proxy.java:397)
            at $Proxy13.copy(Unknown Source)
            at com.ncryptedcloud.cloudprovider.onedrive.OneDriveCloudProvider.copy(OneDriveCloudProvider.java:731)
            at com.ncryptedcloud.operation.cloudprovider.CopyFileOperation.execute(CopyFileOperation.java:72)
            at com.foxykeep.datadroid.service.RequestService.onHandleIntent(RequestService.java:145)
            at com.foxykeep.datadroid.service.MultiThreadedIntentService$IntentRunnable.run(MultiThreadedIntentService.java:170)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:422)
            at java.util.concurrent.FutureTask.run(FutureTask.java:237)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
            at java.lang.Thread.run(Thread.java:818)

[nfuller]

As a workaround, with HttpURLConnection at least, you can set Accept-Encoding: gzip yourself, and then you'd need to deal with wrapping the InputStream in a GZIPInputStream yourself. Not so sure about retrofit / other APIs.

I'd say this is a problem with the OneDrive server. Isn't a minimum gzip encoded content those 10 bytes? Anything smaller is actually not gzip encoded?

[igokoro]

@nfuller, Not sure about the minimum gzip encoded content length. If it's indeed 10 bytes - will file an issue against OneDrive API.

Workaround is pretty simple: supply Accept-Encoding: identity header with request so that response can't be gzipped. But it would be nice handle this inside OkHttp, because it might really take a lot of time to figure out what is wrong!

[nfuller]

@igokoro I'm only going by what I've seen in the past, but it seems to be backed up by http://en.wikipedia.org/wiki/Gzip#File_format

Yes: identity encoding would work, but I assumed you would want compression.

I was thinking about what OkHttp could or should do here. If dealing with gzip, when content-length == 0, it could pretend that was valid? If content length was > 0 but < 10 it would be sure that was invalid but would it be ok to swallow that? 10 bytes is the minimum, but it doesn't mean those 10 are valid. None of these seem to be things that OkHttp could concern itself with.

okio, on the other hand, could maybe be more informative when the gzip data is invalid. Until now I hadn't noticed that okio had it's own gzip implementation. This isn't a criticism of okio: For example, Android's GZIPIntputStream isn't any more friendly: if < 10 header bytes are present you'll just see an EOFException without explanation, which is just the same as okio.

[swankjesse]

Yeah, this is a bug in your server and they should fix it. Work around by explicitly requesting identity encoding.

[igokoro]

@nfuller, @swankjesse got your point. Let's see what OneDrive backend developers will respond.

[rfc2822]

I have the same problem with another server.

According to HTTP/1.1 Gzip Coding, the content is compressed with gzip, which is described in RFC 1952. This document says:

2.2. File format

 A gzip file consists of a series of "members" (compressed data
 sets).  The format of each member is specified in the following
 section.  The members simply appear one after another in the file,
 with no additional information before, between, or after them.

2.3. Member format

 Each member has the following structure:

    +---+---+---+---+---+---+---+---+---+---+
    |ID1|ID2|CM |FLG|     MTIME     |XFL|OS | (more-->)
    +---+---+---+---+---+---+---+---+---+---+

As far as I understand this, it means that gzipped content consists of a series of chunks (members), which have a 10 octet long header. So, it seems to be possible that there are zero members, in which case the content has zero octets.

Even if a series would really mean a series with at least one member, I guess it would increase compatibility to allow zero-length gzip responses and do no harm.

What do you think? Are you sure that there must be at least one member (i.e. at least 10 octets), and if yes, wouldn't it be robust implementation to accept responses with zero members and treat them as what they are, namely empty responses?

[swankjesse]

The gunzip tool doesn’t like an empty file:

$ touch file.gz
$ gunzip file.gz
gunzip: file.gz: unexpected end of file

[rfc2822]

The gunzip tool doesn’t like an empty file:

$ touch file.gz
$ gunzip file.gz
gunzip: file.gz: unexpected end of file

This was also the first thing I have tried :) Although, I wouldn't say that this is normative, especially as RFC 1952 just talks about a series and there are implementations (for instance, above-mentioned servers, or this one) which interpret the standard so that zero chunks are possible. Following the robustness principle (be compatible in what you receive and strict in what you send), I suggest that 0-byte or multi-chunk gzip responses should be accepted.

Am I right that GzipSource in okio is the right place to handle that? As far as I can see, this class doesn't process chunks like described in RFC 1952, but assumes that there is only one chunk with exactly one header, one body and one trailer. If I understand it correctly, this could be a real problem with large responses with more than one gzip member.

I guess okio would be a better place to talk about that?

[swankjesse]

Let's discuss here:

#3759 (comment)