[PkgServer] use Tar.jl to extract tarballs #34
Conversation
src/resource.jl
Outdated
| @@ -21,6 +21,9 @@ const resource_re = Regex(""" | |||
| """, "x") | |||
| const hash_part_re = Regex("/($hash_re)\$") | |||
|
|
|||
| compress(io::IO) = TranscodingStream(GzipCompressor(level=9), io) | |||
src/resource.jl
Outdated
| @@ -21,6 +21,9 @@ const resource_re = Regex(""" | |||
| """, "x") | |||
| const hash_part_re = Regex("/($hash_re)\$") | |||
|
|
|||
| compress(io::IO) = TranscodingStream(GzipCompressor(level=9), io) | |||
| decompress(io::IO) = TranscodingStream(GzipDecompressor(), io) | |||
There was a problem hiding this comment.
No need to involve TranscodingStreams. GzipDecompressorStream can be used directly.
There was a problem hiding this comment.
Can you give an example of how that would be done? I've only used it via TranscodingStreams so far...
There was a problem hiding this comment.
Without checking I'm fairly certain it's exactly the same under the hood, you can just avoid a direct dependency on TranscodingStreams.
|
Similar to https://github.com/JuliaPackaging/PkgServer.jl/pull/29/files#r417042956, there seems also a significant performance issue here. function extract_tar_julia(tarball, outdir)
open(tarball) do io
Tar.extract(decompress(io), outdir)
end
end
function extract_tar(tarball, outdir)
mkpath(outdir)
run(`tar -C $outdir -zxf $tarball`)
end
# tarball is a file of size 3M
@time extract_tar_julia(tarball, "tmp") # 0.06s
@time extract_tar(tarball, "tmp") # 0.03sIf the performance regression is accepted during building a storage server, that during serving contents via pkg server would typically harm user experiences. Perhaps this PR is not wanted at present. |
|
This only happens on the first hit for a given tarball, so I'm not too worried about speed—the first person to get this is going to have a slow experience anyway. All the ones after that will be fast. We can, in principle, verify tarballs without unpacking them (they do need to be uncompressed), but that would need to be implemented. |
|
|
As a part of 3aa8ea3 I actually ended up using the within-tarball tree hash calculation that Stefan implemented in Thanks for the work here @johnnychen94! |
cont. #29