Race condition in 0.9

[palday]

I will try to provide a nicer MWE later, but I ran into a problem while working on some proprietary data. The file in question parses without problem in CSV 0.8.3, but fails under the same options in 0.9. Adding ntasks=1 in CSV 0.9 solves the issue, which makes me suspect a race condition.

cc: @dmbates

[quinnj]

Thanks for reporting; even if you can only share the error + stacktrace you're seeing, that can often be very helpful.

[palday]

Filename passed as a string; kwargs:

comment="#", 
delim=';', 
missingstring=["NA",""],
downcast=true,
drop=[1,2],
decimal=','

Error + stacktrace:

TaskFailedException

nested task error: AssertionError: task_col.column isa Vector{UInt32}

Stacktrace:

[1] syncrefs!(#unused#::Type{String}, col::CSV.Column, task_col::CSV.Column, task_rows::Int64)

@ CSV ~/.julia/packages/CSV/eB98W/src/file.jl:474

[2] multithreadpostparse(ctx::CSV.Context, ntasks::Int64, pertaskcolumns::Vector{Vector{CSV.Column}}, rows::Vector{Int64}, rowchunkguess::Int64, finalrows::Int64, j::Int64, col::CSV.Column)

@ CSV ~/.julia/packages/CSV/eB98W/src/file.jl:425

[3] (::CSV.var"#26#31"{CSV.Context, Vector{Int64}, Vector{Vector{CSV.Column}}, Int64, Int64, CSV.Column, Int64, Int64})()

@ CSV ./threadingconstructs.jl:169

...and 1 more exception.

    sync_end(::Channel{Any})@task.jl:369
    macro expansion@task.jl:388[inlined]
    CSV.File(::CSV.Context, ::Bool)@file.jl:257
    File@file.jl:223[inlined]
    #File#[email protected]:219[inlined]
    top-level scope@Local: 2[inlined]

[quinnj]

Thanks that's helpful. If there's anyway to share the file privately with me, that would be extremely helpful in terms of debugging this; happy to jump on a call/zoom to facilitate sharing if possible. Otherwise, I'll try to walk through the code and figure out how this is getting triggered. In the mean time, you can avoid the error by doing pool=false, which will avoid this code path entirely.

[palday]

Got permission to share:
https://github.com/RePsychLing/SMLP2021datasets/blob/main/Sinika_Timme/data/hexaff_all.csv

Code using that data: https://github.com/RePsychLing/SMLP2021datasets/blob/1b2d1fa4842778574586359d257a9931349b7f79/instructors/Sinika_Timme/analysis.jl#L51-L59

[quinnj]

I could reproduce the issue on 0.9.1 and the proposed fix in #898 fixes it for me.

[palday]

In the REPL the fix works, but it in Pluto it doesn't:

TaskFailedException

nested task error: TypeError: in typeassert, expected Dict{Union{Missing, String}, UInt32}, got a value of type Dict{Union{Missing, WeakRefStrings.String31}, UInt32}

Stacktrace:

[1] syncrefs!(#unused#::Type{String}, col::CSV.Column, task_col::CSV.Column, task_rows::Int64)

@ CSV ~/.julia/packages/CSV/h66c0/src/file.jl:476

[2] multithreadpostparse(ctx::CSV.Context, ntasks::Int64, pertaskcolumns::Vector{Vector{CSV.Column}}, rows::Vector{Int64}, rowchunkguess::Int64, finalrows::Int64, j::Int64, col::CSV.Column)

@ CSV ~/.julia/packages/CSV/h66c0/src/file.jl:422

[3] (::CSV.var"#26#31"{CSV.Context, Vector{Int64}, Vector{Vector{CSV.Column}}, Int64, Int64, CSV.Column, Int64, Int64})()

@ CSV ./threadingconstructs.jl:169

    sync_end(::Channel{Any})@task.jl:369
    macro expansion@task.jl:388[inlined]
    CSV.File(::CSV.Context, ::Bool)@file.jl:257
    File@file.jl:223[inlined]
    #File#[email protected]:219[inlined]
    top-level scope@Local: 2[inlined]

Note that Pluto runs the notebooks in a Distributed worker process, so you get slightly different scheduling. I think the problem here could be sidestepped by specifying the string type, but the auto detection threshold is what has the race condition.

[quinnj]

Thanks for the follow up; I'll try to dig in and see if there's still a promotion problem with the refpool sometimes.