panic: cannot consume from pending buffer

[Turbo87]

In rust-lang/crates.io#9622 we upgraded crates.io to the latest async-compression release (v0.4.14), which appears to be causing issues on our staging environment:

https://rust-lang.sentry.io/share/issue/9e273e28600f48a5aec2ad9e127afbbf/

The code inside the object_store crate is throwing a "cannot consume from pending buffer" panic, but the root cause seems to be originating from the async-compression crate. We use both crates roughly like this:

let reader = object_store::buffered::BufReader::new(store, &meta);
let decompressor = Decompressor::from_extension(reader, path.extension())?;

I assume that #294 is causing this issue, but I don't know the code well enough to find the exact bug in the code changes that are causing this behavior.

[Turbo87]

@robjtede it looks like #299 didn't fix the problem, and GitHub won't let me reopen this issue 😅

[NobodyXu]

@Turbo87 Which compression algorithm are you using?

I suspect the decoder impl is wrong.

[Turbo87]

Which compression algorithm are you using?

zstd and gzip (see https://github.com/rust-lang/crates.io/blob/51c3f84456ecc1fc6131a1f82478536e05e4569d/crates/crates_io_cdn_logs/src/compression.rs). I just checked, we're seeing the issue for both of them apparently.

[NobodyXu]

Thanks, I will have to take a look at them.

Surely I have missed something.