precompile shouldn't try to overwrite stale read-only cache files

[stevengj]

A lot of people are seeing failures on JuliaBox (JuliaCloud/JuliaBox#338, JuliaPy/PyPlot.jl#185, JuliaLang/IJulia.jl#159) because JuliaBox ships precompiled cache files in a read-only directory /opt/julia_packages/.julia/lib/v0.4/. If these cache files are stale, Julia tries to recompile them, but it begins by trying to delete the stale old cache file, which throws an exception, e.g.

julia> using Compat                                                                                                                                                
INFO: Recompiling stale cache file /opt/julia_packages/.julia/lib/v0.4/Compat.ji for module Compat.                                                                
ERROR: unlink: read-only file system (EROFS)                                                                                                                       
 in unlink at fs.jl:102                                                                                                                                            
 in rm at file.jl:59                                                                                                                                               
 in create_expr_cache at loading.jl:330                                                                                                                            
 in recompile_stale at loading.jl:461                                                                                                                              
 in _require_from_serialized at loading.jl:83                                                                                                                      
 in _require_from_serialized at ./loading.jl:109                                                                                                                   
 in require at ./loading.jl:219 

[stevengj]

This seems to have been partially worked around in #14145 by @eschnett, but it looks like it will still try to write to the read-only location.

[stevengj]

Probably the recompile_stale function should call create_expr_cache(path, joinpath(Base.LOAD_CACHE_PATH[1], basename(cachefile))) so that the new cache file is always created in the default location (the head of the path), not where the old cache file was found.

[tanmaykm]

Would it be better to just check for write permissions and skip updating the cache file if it is readonly? It seems to fall back to the default location Base.LOAD_CACHE_PATH[1] in that case.

I tried with the following change in loading.jl.

function recompile_stale(mod, cachefile)
    if ((stat(cachefile).mode & 0o222) == 0)
        warn("could not remove stale cache file $cachefile")
        return
    end
...
end

[eschnett]

Checking file permissions isn't a reliable way to determine whether a file can be deleted. For example, in Unix you need write permissions to the directory, and on a network file system or with access control lists, it's up to the file server anyway. It is usually easier to just try to delete the file, and check the error condition afterwards.

[tanmaykm]

Ah yes. So, then may be a try...catch?

[stevengj]

Even if it is writable, I don't think we should be writing new cache files to arbitrary locations in the path. I think it is better to always write new files to Base.LOAD_CACHE_PATH[1], which will normally be in ~/.julia/lib. (See #14369.)

e.g. if someone ships me a .ji file that I stick into a directory somewhere, I don't think that Julia should overwrite it even if it is in the path — I want my files touched only if I specifically request it; it is more desirable if all of our writing is confined to .julia. Writing automatically to the filesystem should be viewed as a dangerous, undesirable activity that should be confined to as few well-defined directories as possible.

[tanmaykm]

Ok, yes that sounds reasonable.

[jonasaugust]

I'm also getting the "read-only file system" error when trying to add PyPlot to my JuliaBox instance with kernel 0.4.2. Although this issue has been closed in December... I'm guessing it hasn't been installed to the JuliaBox server yet. How/when does that happen, and how would we come to know it when it does?
(And, of course, thanks for all your work on Julia!)

[tkelman]

#14369 was backported to 0.4.3, so juliabox should just need an upgrade to that.

[jonasaugust]

@tkelman Thanks for making it clearer.