Support for single-file multithreading

[mbauman]

#230 implemented thread-safety at a file level. However, if I try to read from a single file from multiple threads, I hit segfaults. I understand that multithreaded writing to a single file is fraught, but would it be a major challenge to support multithreaded reads?

[JonasIsensee]

Hi @mbauman,

this really should be possible.
First, the reason that it doesn't work:
JLD2 (globally) keeps track of all files that are being opened.
This is to ensure that you e.g. don't accidentally open a single file in both read and write mode at the same time.

• If you disable that logic, threaded reading should work. see

• JLD2.jl/src/JLD2.jl

  Line 272 in e746c89

  if haskey(OPEN_FILES, rname)

With the current logic it doesn't work since the same file handle is being used to process reads. And also the JLDFile object has plenty internal caching that could hit race-conditions.

• This could in principle be fixed by introducing locks around all operations that change the cache or really read from the file.

[ejmeitz]

How hard would it be for a non-JLD expert to implement this? I have an application where everything threads perfectly but a HDF5 read is the thing preventing me from multi-threading the program. I can change my program to get around this but it would make my code a lot more hacky.

[JonasIsensee]

This wouldn't be very hard and does not require any real JLD2 knowledge.

This is the relevant function definition:

JLD2.jl/src/JLD2.jl

Line 254 in e746c89

function jldopen(fname::AbstractString, wr::Bool, create::Bool, truncate::Bool, iotype::T=MmapIO;

Currently, there is a "global" list that keeps track of all open files and ensures that only a single JLDFile can exist per file.

This condition could be relaxed:

• Always create a new file handle for read-only access
• Keep a separate list of "read-only" file handles (threadsafe). (To throw an error if you try to also create a write access file handle while a read-only one is open)
• Alternatively introduce a new optional kwarg that will bypass the whole safety mechanism

[JonasIsensee]

implemented by #477