Piracy in the StdLibs

[oxinabox]

I have been playing with the idea of automatically detecting type piracy.
https://discourse.julialang.org/t/pirate-hunter/20402

Here is the results of running it will the standard libraries loaded.

• [1] BigFloat(::Irrational{:SQRT_HALF}) in Random at irrationals.jl:158
• [2] (T::Union{Type{Int8}, Type{UInt8}})(c::Char) in Base at char.jl:170
• [3] Float64(::Irrational{:SQRT_HALF}) in Random at irrationals.jl:166
• [4] Float32(::Irrational{:SQRT_HALF}) in Random at irrationals.jl:167
• [5] (T::Union{Type{Int8}, Type{UInt8}})(c::Char) in Base at char.jl:170
• [6] apropos(io::IO, needle::Regex) in REPL at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/REPL/src/docview.jl:621
• [7] apropos(io::IO, string) in REPL at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/REPL/src/docview.jl:619
• [8] apropos(string) in REPL at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/REPL/src/docview.jl:618
• [9] doc(binding::Base.Docs.Binding) in REPL at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/REPL/src/docview.jl:85
• [10] doc(obj::UnionAll) in REPL at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/REPL/src/docview.jl:129
• [11] doc(object) in REPL at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/REPL/src/docview.jl:130
• [12] formatdoc(d::Base.Docs.DocStr) in REPL at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/REPL/src/docview.jl:58
• [13] formatdoc(buffer, d, part) in REPL at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/REPL/src/docview.jl:64
• [14] parsedoc(d::Base.Docs.DocStr) in REPL at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/REPL/src/docview.jl:67
• [15] adjoint(B::Union{BitArray{1}, BitArray{2}}) in LinearAlgebra at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/LinearAlgebra/src/bitarray.jl:223
• [16] fetch(x) in Distributed at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Distributed/src/remotecall.jl:533
• [17] in(v::VersionNumber, r::VersionNumber) in Pkg.Types at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Pkg/src/versions.jl:141
• [18] isless(a::Base.UUID, b::Base.UUID) in Pkg.Types at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Pkg/src/Types.jl:38
• [19] kron(a::BitArray{1}, b::BitArray{1}) in LinearAlgebra at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/LinearAlgebra/src/bitarray.jl:96
• [20] kron(a::BitArray{2}, b::BitArray{2}) in LinearAlgebra at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/LinearAlgebra/src/bitarray.jl:108
• [21] kron(a::Number, b::Union{Number, Union{AbstractArray{T,1}, AbstractArray{T,2}} where T}) in LinearAlgebra at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlid/v1.1/LinearAlgebra/src/dense.jl:359
• [22] rand(X) in Random at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Random/src/Random.jl:224
• [23] show(io::IO, ::MIME{Symbol("text/csv")}, a) in DelimitedFiles at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/DelimitedFiles/src/DelimitedFiles.jl:828
• [24] show(io::IO, ::MIME{Symbol("text/tab-separated-values")}, a) in DelimitedFiles at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/DelimitedFiles/src/DelimitedFiles.jl:829
• [25] string(x::Array{String,1}) in Pkg.Types at /Users/osx/buildbot/slave/package_osx64/build/usr/share (Pkg does a nonsensical type piracy. Pkg.jl#1034)
  /julia/stdlib/v1.1/Pkg/src/Types.jl:1453
• [26] transpose(B::Union{BitArray{1}, BitArray{2}}) in LinearAlgebra at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/LinearAlgebra/src/bitarray.jl:224
• [27] wait(fd::RawFD) in FileWatching at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/FileWatching/src/FileWatching.jl:466

I'm not sure how many of these already have issues, eg #28234
and how many are harmless.
I am sure someone more knowledge-able than me can strike a bunch of them as not problematic.
But I figured I should share the list

[JeffBezanson]

Wow, a couple interesting items here.

(T::Union{Type{Int8}, Type{UInt8}})(c::Char) in Base at char.jl:170

That's entirely in base, seems like an incorrect result.

string(x::Vector{String}) = x

Pkg, are you kidding me? It doesn't even return a string...

• fetch(x) in Distributed

Yep, should be moved to Base. Not sure fetch should be defined on Any, but that's a separate question.

It looks like every useful method of kron is in LinearAlgebra, so that's not so bad.

wait(::RawFD) is ok since it strictly adds functionality.

[oxinabox]

You have the power to tick them (and other meanless ones and false positives) off right?

[timholy]

That is really cool. I worry about about people getting too worried about piracy (there are occasionally some very good reasons for it), but having a tool to detect seems like a clear win.

[KristofferC]

Pkg, are you kidding me? It doesn't even return a string...

JuliaLang/Pkg.jl#1034

[JeffBezanson]

Right; I look forward to wearing my "I'm a type pirate" T-shirt :) I think this is best seen as a heuristic for finding misplaced code, unintentional method extensions, etc.

[JeffBezanson]

• BigFloat(::Irrational{:SQRT_HALF}) in Random at irrationals.jl:158

This is kind of OK since it's the only way to make new irrational numbers. However, we should perhaps add a couple more like sqrt(2) to Base.

• apropos(io::IO, needle::Regex) in REPL at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/REPL/src/docview.jl:621

This is intentional, since we wanted to move doc viewing code out of Base. But I think it's still up in the air where all the Base.Docs code should go exactly.

• in(v::VersionNumber, r::VersionNumber)

Could maybe go in base, but VersionNumber isn't iterable. Maybe Pkg could arrange to wrap the argument in a VersionRange instead?

• isless(a::Base.UUID, b::Base.UUID)

Can move to Base.

• show(io::IO, ::MIME{Symbol("text/csv")}, a)

This might actually make sense in Base as well. The code for writing is much simpler than the code for parsing. We should maybe even use this as the output format of print on matrices. It should probably also be specialized at least as a::AbstractArray.

[ararslan]

in(v::VersionNumber, r::VersionNumber)

JuliaLang/Pkg.jl#1035

[00vareladavid]

[25] string(x::Array{String,1}) in Pkg.Types

Fix at JuliaLang/Pkg.jl#1036

[ararslan]

• isless(a::Base.UUID, b::Base.UUID)

Can move to Base.

#30947

[dlfivefifty]

There's also vcat(::Vector...) in SparseArrays.jl:

julia/stdlib/SparseArrays/src/sparsevector.jl

Line 1072 in 9bd5c18

vcat(A::Vector...) = Base.typed_vcat(promote_eltype(A...), A...)

SparseArrays seems like a chronic offender: #32213

[KristofferC]

This is missing e.g

julia> @which rand(5,5) * rand(5,5)
*(A::AbstractArray{T,2} where T, B::AbstractArray{T,2} where T) in LinearAlgebra at ...julia/stdlib/v1.3/LinearAlgebra/src/matmul.jl:152

[KristofferC]

I worry about about people getting too worried about piracy (there are occasionally some very good reasons for it), but having a tool to detect seems like a clear win.

FWIW, type piracy means that it is impossible to look at the Project and know what stdlibs a package actually use. For example, you can do matrix multiplication or call rand() without requiring LinearAlgebra or Random in your Project file. In the scenario where one wants to ship an app with a custom sysimage it would be desirable to exclude the stdlibs not needed by the app (for smaller file size and faster load times). Type-piracy now comes and puts a big stick in the wheel because there is no way to know what stdlibs the app actually need because the code might be filled with matrix multiplication but no trace of LinearAlgebra is found.

[ViralBShah]

#43127 fixes some of the sparse arrays related ones.