Lots of allocations related to buffering

[kcajf]

julia> data = rand(Float32, 1_000_000);

julia> io = open("foo", "w");

julia> zio = CodecZstd.ZstdCompressorStream(io);

julia> @time write(zio, data)
  0.023578 seconds (60.39 k allocations: 2.623 MiB)
4000000

julia> @time write(zio, data)
  0.024188 seconds (60.39 k allocations: 2.623 MiB)
4000000

julia> zio = CodecZstd.ZstdCompressorStream(io, bufsize=500_000);

julia> @time write(zio, data)
  0.011804 seconds (1.02 k allocations: 44.641 KiB)
4000000

julia> @time write(zio, data)
  0.013522 seconds (1.16 k allocations: 51.016 KiB)
4000000

julia> zio = CodecZstd.ZstdCompressorStream(io, bufsize=5_000_000);

julia> @time write(zio, data)
  0.002285 seconds (1 allocation: 16 bytes)
4000000

julia> @time write(zio, data)
  0.015165 seconds (146 allocations: 6.391 KiB)
4000000

julia> @time write(zio, data)
  0.017407 seconds (146 allocations: 6.391 KiB)
4000000

This is a very simple MWE of a typical use case for CodecZstd - writing to a file. It seems to cause an unexpectedly large amount of allocations. They seem to be related to the buffer size, presumably a bug in the growth or re-use of buffers.

[nhz2]

This issue seems to have been fixed at some point.

Let me know if this is still a problem and I will reopen the issue.

julia> data = rand(Float32, 1_000_000);

julia> io = open(tempname(), "w");

julia> zio = CodecZstd.ZstdCompressorStream(io);

julia> @time write(zio, data)
  0.006532 seconds (1 allocation: 16 bytes)
4000000

julia> @time write(zio, data)
  0.006255 seconds (1 allocation: 16 bytes)
4000000

julia> zio = CodecZstd.ZstdCompressorStream(io, bufsize=500_000);

julia> @time write(zio, data)
  0.004821 seconds (1 allocation: 16 bytes)
4000000

julia> @time write(zio, data)
  0.005476 seconds (1 allocation: 16 bytes)
4000000

julia> zio = CodecZstd.ZstdCompressorStream(io, bufsize=5_000_000);

julia> @time write(zio, data)
  0.001108 seconds (1 allocation: 16 bytes)
4000000

julia> @time write(zio, data)
  0.006812 seconds (1 allocation: 16 bytes)
4000000

julia> @time write(zio, data)
  0.007056 seconds (1 allocation: 16 bytes)
4000000

(jl_1fF2J3) pkg> st -m
Status `/tmp/jl_1fF2J3/Manifest.toml`
  [6b39b394] CodecZstd v0.8.2
  [692b3bcd] JLLWrappers v1.5.0
  [21216c6a] Preferences v1.4.3
  [3bb67fe8] TranscodingStreams v0.10.8
  [3161d3a3] Zstd_jll v1.5.6+0
  [56f22d72] Artifacts
  [ade2ca70] Dates
  [8f399da3] Libdl
  [de0858da] Printf
  [fa267f1f] TOML v1.0.3
  [4ec0a83e] Unicode