-
-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
trouble with filter(!ismissing, x) #27809
Comments
To deal with the immediate problem, we could define a special case
|
This gives me a fairly reproducible error (not crash), usually in the third iteration, occasionally later.
The error looks like this:
|
And this minor variation consistently crashes within 2-7 iterations.
Basically the same backtrace as before:
|
This is reproducible here: julia> b = Vector{Union{Missing, Float64}}()
0-element Array{Union{Missing, Float64},1}
julia> while length(b) < 2^17
push!(b, 0.0)
end
julia> any(ismissing, b)
false
julia> push!(b, 0.0);
julia> any(ismissing, b)
true
julia> findall(ismissing, b)
8-element Array{Int64,1}:
131065
131066
131067
131068
131069
131070
131071
131072 |
Taking a look |
The performance issue is caused by O(n^2) time in |
Split off performance improvement to #27825 |
…g realloc was causing problems for isbits Union arrays, which have different layout requirements than regular arrays
With
I can no longer reproduce the Illegal instruction but instead get another
with
|
…r jl_array_t* with potential a->offset and a->maxsize values. Before, any resizing operations on isbits Union arrays (push, pushfirst, insert, append, deleteat, slice, etc.) incurred extra cost by having to constantly move the type tag bytes to be directly after the last array element. By treating them rather as jl_array_t* objects, sharing the a->offset and a->maxsize fields with the parent array, resizing operations now only require moving bytes around if the parent array's data itself must be moved. Fixes #27825 and #27809.
…r jl_array_t* with potential a->offset and a->maxsize values. Before, any resizing operations on isbits Union arrays (push, pushfirst, insert, append, deleteat, slice, etc.) incurred extra cost by having to constantly move the type tag bytes to be directly after the last array element. By treating them rather as jl_array_t* objects, sharing the a->offset and a->maxsize fields with the parent array, resizing operations now only require moving bytes around if the parent array's data itself must be moved. Fixes #27825 and #27809.
…r jl_array_t* with potential a->offset and a->maxsize values. Before, any resizing operations on isbits Union arrays (push, pushfirst, insert, append, deleteat, slice, etc.) incurred extra cost by having to constantly move the type tag bytes to be directly after the last array element. By treating them rather as jl_array_t* objects, sharing the a->offset and a->maxsize fields with the parent array, resizing operations now only require moving bytes around if the parent array's data itself must be moved. Fixes JuliaLang#27825 and JuliaLang#27809.
@quinnj It seems this is fixed now? |
Yes, this is fixed now. |
Summary:
filter(!ismissing, x)
gives wrong results, is exceedingly slow, and occasionally leads to crashes. I would suspect code generation goes wrong.As an accidental spinoff from #27781 I ran into this, which gives unexpected results and slowness.
Increasing the size from one million to ten million elements increases the running time from 8 seconds to 2700 seconds. Moreover I've had Julia crashing on me twice while experimenting with this, including this complete session:
The text was updated successfully, but these errors were encountered: