Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stream: fix reading LibuvStream into array #56092

Merged
merged 1 commit into from
Oct 18, 2024
Merged

stream: fix reading LibuvStream into array #56092

merged 1 commit into from
Oct 18, 2024

Conversation

vtjnash
Copy link
Member

@vtjnash vtjnash commented Oct 10, 2024

Adds a new abstraction take!(dst::Array{T,N}, src::Array{T,N}) for doing an efficient copyto! equivalent. Previously it was assumed that compact did this automatically, which wasn't a great assumption.

Fixes #56078

@vtjnash vtjnash added io Involving the I/O subsystem: libuv, read, write, etc. backport 1.11 Change should be backported to release-1.11 labels Oct 10, 2024
@giordano giordano added the needs tests Unit tests are required for this change label Oct 10, 2024
@nhz2
Copy link
Contributor

nhz2 commented Oct 10, 2024

This is still broken.

readbytes! is documented to never decrease the size of the input vector, but in this example, it gets changed from 1024 to 10.

function mwe(filepath, nb, max_nb)
    write(filepath, ones(UInt8, nb))
    p = open(`cat $filepath`; read=true)
    b = Vector{UInt8}(undef, 1024)
    nr = readbytes!(p, b, max_nb)
    return nr, b
end
nr, b = mwe(tempname(), 10, 10000)
(10, UInt8[0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01])

@ararslan
Copy link
Member

Adds a new abstraction take!(dst::Array{T,N}, src::Array{T,N}) for doing an efficient copyto! equivalent.

I don't understand how this is intended to differ meaningfully from copyto!. Also, isn't this adding a new feature (in the form of a new method for an exported function) that would need to be backported to a patch release in order to fix the regression in 1.11?

@vtjnash vtjnash added merge me PR is reviewed. Merge when all tests are passing and removed needs tests Unit tests are required for this change labels Oct 15, 2024
@ararslan
Copy link
Member

I'm confused, the "needs tests" label was removed but tests weren't added. It would be good to include a test to ensure #56078 is indeed fixed and to prevent it from regressing in the future.

Since this is adding methods to an exported function, it would be good to document the methods. If they aren't intended to be user-facing, perhaps make it a separate, internal-only function?

@vtjnash
Copy link
Member Author

vtjnash commented Oct 16, 2024

I wasn't sure about what it needed for a new test here, and didn't seem worth spending time on it, while the followup issues was already caught by the existing read tests. You are right that these new functions would be internal-only, by policy, for v1.11 so they should get documented separately on master. But assuming these are the names we like, then they can already be added internally like this.

@KristofferC
Copy link
Member

Why can't these just be internal functions over extending take!? Keep the bugfix separate from any new API.

@ararslan
Copy link
Member

I wasn't sure about what it needed for a new test here

Would it work to use the test case from #56078? Just guessing but this might do it:

function issue56078(filepath, nb, max_nb)
    write(filepath, ones(UInt8, nb))
    @static if Sys.iswindows()
        cmd = `cmd.exe /C type $filepath`
    else
        cmd = `cat $filepath`
    end
    p = open(cmd; read=true)
    b = Vector{UInt8}(undef, 1024)
    nr = readbytes!(p, b, max_nb)
    return nr, b
end
mktemp() do file, io
    close(io)
    nr, v = issue56078(file, 53209, typemax(Int))
    @test nr == 53209
    @test length(v) > 1024
end

Why can't these just be internal functions over extending take!?

We could add methods to take! later that would simply call the internals added in service of this issue.

@vtjnash
Copy link
Member Author

vtjnash commented Oct 17, 2024

I couldn't think of a better name, since they do what take! does, unless take!! is somehow better since we mutate both arguments?

@KristofferC
Copy link
Member

KristofferC commented Oct 18, 2024

Call it _take! then. Whatever to decouple the bugfix from the new feature.

Adds a new abstraction `take!(::Array{T,N}, ::Array{T,N})` for doing an
efficient `copyto!` equivalent. Previously it was assumed that `compact`
did this automatically, which wasn't a great assumption.

Fixes #56078
@vtjnash vtjnash merged commit fc40e62 into master Oct 18, 2024
4 of 7 checks passed
@vtjnash vtjnash deleted the jn/56078 branch October 18, 2024 19:33
@giordano giordano removed the merge me PR is reviewed. Merge when all tests are passing label Oct 18, 2024
giordano pushed a commit that referenced this pull request Oct 19, 2024
Adds a new internal function `_take!(dst::Array{T,N}, src::Array{T,N})` for
doing an efficient `copyto!` equivalent. Previously it was assumed that
`compact` did this automatically, which wasn't a great assumption.

Fixes #56078

(cherry picked from commit fc40e62)
@giordano giordano removed the backport 1.11 Change should be backported to release-1.11 label Oct 19, 2024
KristofferC pushed a commit that referenced this pull request Oct 21, 2024
Adds a new internal function `_take!(dst::Array{T,N}, src::Array{T,N})` for
doing an efficient `copyto!` equivalent. Previously it was assumed that
`compact` did this automatically, which wasn't a great assumption.

Fixes #56078
KristofferC added a commit that referenced this pull request Nov 21, 2024
Backported PRs:
- [x] #55886 <!-- irrationals: restrict assume effects annotations to
known types -->
- [x] #55867 <!-- update `hash` doc string: `widen` not required any
more -->
- [x] #56084 <!-- slightly improve inference in precompilation code -->
- [x] #56088 <!-- make `Base.ANSIIterator` have a concrete field -->
- [x] #54093 <!-- Fix `JULIA_CPU_TARGET` being propagated to workers
precompiling stdlib pkgimages -->
- [x] #56165 <!-- Fix markdown list in installation.md -->
- [x] #56148 <!-- Make loading work when stdlib deps are missing in the
manifest -->
- [x] #56174 <!-- Fix implicit `convert(String, ...)` in several places
-->
- [x] #56159 <!-- Add invalidation barriers for `displaysize` and
`implicit_typeinfo` -->
- [x] #56089 <!-- Call `MulAddMul` instead of multiplication in
_generic_matmatmul! -->
- [x] #56195 <!-- Include default user depot when JULIA_DEPOT_PATH has
leading empty entry -->
- [x] #56215 <!-- [REPL] fix lock ordering mistake in load_pkg -->
- [x] #56251 <!-- REPL: run repl hint generation for modeswitch chars
when not switching -->
- [x] #56092 <!-- stream: fix reading LibuvStream into array -->
- [x] #55870 <!-- fix infinite recursion in `promote_type` for
`Irrational` -->
- [x] #56227 <!-- Do not call `rand` during sysimage precompilation -->
- [x] #55741 <!-- Change annotations to use a NamedTuple -->
- [x] #56149 <!-- Specialize adding/subtracting mixed
Upper/LowerTriangular -->
- [x] #56214 <!-- fix precompile process flags -->
- [x] #54471
- [x] #55622
- [x] #55704
- [x] #55764
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
io Involving the I/O subsystem: libuv, read, write, etc.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

readbytes!(::Process, ::Vector{UInt8}, ::Int) does not read bytes on 1.11
5 participants