Zlib::Reader: Reading to EOF deadlocks application

[z64]

Zlib::Reader can deadlock the application if you read to EOF:

require "zlib"

spawn do
  i = 0
  loop do
    puts "[#{i}] looping"
    i += 1
    sleep 1
  end
end

# Wait 3 sec to start
sleep 3

File.open("data.bin", "r") do |file|
  Zlib::Reader.new(file).gets_to_end
end

$ time cr run.cr
[0] looping                       
[1] looping                                      
[2] looping                                
^C                                                  
real    0m9.508s                      
user    0m3.162s                      
sys     0m3.205s

I've created https://github.com/z64/zlib-freeze which contains a sample data blob and the above code that demonstrates the deadlock, as well as an excerpt from lldb in README.md.

Background

I found this while implementing business logic of a websocket protocol that sends chunks of zlib-compressed data over multiple frames, where you buffer each frame received until a "end of flush" binary suffix is read; so you must use a single instance of Zlib::Reader throughout your connections lifetime over multiple frames. Trying to inflate a single blob (shown in the above repo) deadlocked the app.

I suspect the correct behavior here should be to raise an exception; as a workaround I'm infalting into a slice (reader.read(slice)) which will not exhibit this deadlock behavior.

[kostya]

when i use my static crystal analyze tool, i find one bug in zlib flate code, not sure it related here or not, or how to test it (just want to note it somewhere).
https://github.com/crystal-lang/crystal/blob/master/src/flate/reader.cr#L97

          ret = LibZ.inflateSetDictionary(pointerof(@stream), dict, dict.size)
          next if ret == LibZ::Error::OK

ret it Int, and compared with enum, this is always false.

[z64]

Interesting! Unfortunately I don't think that applies here, at least I'm pretty sure - I monkey patched read to correct that comparison, as well as a debug p! ret == LibZ::Error::OK.value before that line, and nothing was printed, and the example application from my OP still deadlocks.

[rdp]

wonder if related to #4650 ...

[z64]

Good call, maybe! 🙂

Reviewing the last few comments it seems like it may be, though the lldb inspections seem to be a bit different. Though what @RX14 said last in that thread sounds plausible. Either way, my zlib-freeze repo 100% reproduces (at least) this issue.. we just need someone who knows how to debug this further. I don't know where to go from here, but happy to dig further if someone points me in the right direction.