Error when loading old file during "Upgrading JLSO format from v2 to v3"

[oxinabox]

Originally posted by @yakir12 in #32 (comment)

I guess this is related. Anything I can do about it, none of those package were relevant, but yea, when I saved that file they were in the stack. But they aren't needed to parse the data in the file.

julia> using JLSO

julia> data = JLSO.load("trackdata.jlso")
[info | JLSO]: Upgrading JLSO format from v2 to v3
Activating new environment at `/tmp/jl_eLIz6c/Project.toml`
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
[warn | JLSO]: Failed to construct an environment with the provide package version (Dict("Rsvg" => v"0.2.3","Scallops" => v"0.1.0","Bills" => v"0.1.0","Memoize" => v"0.3.0","ImageMorphology" => v"0.2.4","Dungalyze" => v"0.1.0","Juno" => v"0.7.0","OffsetArrays" => v"0.11.1","IdentityRanges" => v"0.3.0","OrderedCollections" => v"1.1.0","Symata" => v"0.4.5","Rematch" => v"0.2.0","ImageFiltering" => v"0.6.4","StatsMakie" => v"0.0.6","AxisArrays" => v"0.3.0","ColorVectorSpace" => v"0.6.2","DataStructures" => v"0.17.0","ImageMagick" => v"0.7.5","CategoricalArrays" => v"0.5.4","GeometryTypes" => v"0.7.5","HCubature" => v"1.4.0+","DungAnalyse" => v"0.1.0","UnitfulAngles" => v"0.5.0","IntervalArithmetic" => v"0.16.0","GtkReactive" => v"0.6.0","DataFrames" => v"0.19.0","SpecialFunctions" => v"0.7.2","PlotlyJS" => v"0.12.5","LowLevelParticleFilters" => v"0.2.0","JuliaDB" => v"0.12.0+","GR" => v"0.40.0","DungRegister" => v"0.1.0","TableView" => v"0.3.0","Distributions" => v"0.20.0","JSON3" => v"0.1.4","ImageDraw" => v"0.2.1","FixedPointNumbers" => v"0.5.3","LabelledArrays" => v"0.7.1","VideoIO" => v"0.6.8","Combinatorics" => v"0.7.0","TextParse" => v"0.9.1","MAT" => v"0.5.0+","HDF5" => v"0.12.0","ColorTypes" => v"0.8.0","CatViews" => v"1.0.0","AbstractPlotting" => v"0.9.8","Makie" => v"0.9.4","LsqFit" => v"0.8.1","jlpkg" => v"1.0.3","Budgets" => v"0.1.0","Polynomials" => v"0.5.2","ProgressMeter" => v"1.0.0","IntervalRootFinding" => v"0.4.0","FFMPEG" => v"0.2.1","FileIO" => v"1.0.7","ImageCore" => v"0.8.4","InteractiveChaos" => v"0.3.2","TypedTables" => v"1.2.0","Cuba" => v"2.0.0","SatelliteToolbox" => v"0.6.2","GLMakie" => v"0.0.7","ApproxFun" => v"0.11.1","IndexedTables" => v"0.12.2","TerminalMenus" => v"0.1.0+","PrettyTables" => v"0.5.0","ExtractFrames" => v"0.1.0","AngleBetweenVectors" => v"0.3.0","CSV" => v"0.5.9","Exintric" => v"0.1.0","ImageSegmentation" => v"1.2.0","GeoJSON" => v"0.4.0","Revise" => v"2.1.6","CleanDung" => v"0.1.0","Databula" => v"0.1.0","StructArrays" => v"0.4.0","Atom" => v"0.8.5","BenchmarkTools" => v"0.4.2","JLSO" => v"1.1.0","LightQuery" => v"0.3.1","Optim" => v"0.19.1","StatsPlots" => v"0.11.0","TimeZones" => v"0.9.1","WeightedOnlineStats" => v"0.3.1","ColorSchemes" => v"3.3.0","PrettyPrint" => v"0.1.0","IntervalTrees" => v"1.0.0","Match" => v"1.0.2","Calibeetle" => v"0.1.0","PyCall" => v"1.91.2","Traceur" => v"0.3.0","DataFramesMeta" => v"0.5.0","JSON" => v"0.21.0","StatsBase" => v"0.30.0","SCS" => v"0.5.1","Images" => v"0.18.0","StringBuilders" => v"0.2.0","Dierckx" => v"0.4.1","DungBase" => v"0.1.0","Plots" => v"0.25.3","LaTeXTabulars" => v"0.1.1","PyPlot" => v"2.8.1","TestImages" => v"0.5.1","VideoPlayer" => v"0.1.0","Unitful" => v"0.16.0","StaticArrays" => v"0.11.0","BinDeps" => v"0.8.10","ExcelFiles" => v"1.0.0","ImageView" => v"0.9.0","Parsers" => v"0.3.6","UnicodePlots" => v"1.1.0","DynamicalBilliards" => v"3.5.6","Mustache" => v"0.5.12","Flatten" => v"0.2.1","Transducers" => v"0.2.1","Observables" => v"0.2.3","Documenter" => v"0.23.0","IntervalSets" => v"0.3.1","TextUserInterfaces" => v"0.1.0","JLD" => v"0.9.1","Interpolations" => v"0.12.2","ControlSystems" => v"0.5.3","Query" => v"0.12.0","PGFPlotsX" => v"1.0.0","MappedArrays" => v"0.2.1","LaTeXStrings" => v"1.0.3","Rebugger" => v"0.3.2","SplitApplyCombine" => v"0.4.1","Tables" => v"0.2.9","RayTraceEllipsoids" => v"0.1.0","Blink" => v"0.10.1","ForceImport" => v"0.0.3","CoordinateTransformations" => v"0.5.0","Colors" => v"0.9.5","Diana" => v"0.1.0","MacroTools" => v"0.5.1","MATLAB" => v"0.7.3","ProfileView" => v"0.4.1","BlockArrays" => v"0.9.1","RecursiveArrayTools" => v"0.20.0","Sobol" => v"1.2.0","DataValues" => v"0.4.12","AWSS3" => v"0.5.0","DataDeps" => v"0.6.4","Rotations" => v"0.11.1","Gtk" => v"0.17.0")): Pkg.Types.PkgError("The following package names could not be resolved:\n * Bills (not found in project, manifest or registry)\n * Budgets (not found in project, manifest or registry)\n * Calibeetle (not found in project, manifest or registry)\n * CleanDung (not found in project, manifest or registry)\n * Databula (not found in project, manifest or registry)\n * Dungalyze (not found in project, manifest or registry)\n * DungAnalyse (not found in project, manifest or registry)\n * DungRegister (not found in project, manifest or registry)\n * Exintric (not found in project, manifest or registry)\n * ExtractFrames (not found in project, manifest or registry)\n * RayTraceEllipsoids (not found in project, manifest or registry)\n * Scallops (not found in project, manifest or registry)\n * VideoPlayer (not found in project, manifest or registry)\nPlease specify by known `name=uuid`.").
 Falling back to simply adding the packages.
Activating environment at `~/coffeebeetlearticle/closednest/Project.toml`
ERROR: The following package names could not be resolved:
 * Bills (not found in project, manifest or registry)
 * Budgets (not found in project, manifest or registry)
 * Calibeetle (not found in project, manifest or registry)
 * CleanDung (not found in project, manifest or registry)
 * Databula (not found in project, manifest or registry)
 * Dungalyze (not found in project, manifest or registry)
 * DungAnalyse (not found in project, manifest or registry)
 * DungRegister (not found in project, manifest or registry)
 * Exintric (not found in project, manifest or registry)
 * ExtractFrames (not found in project, manifest or registry)
 * RayTraceEllipsoids (not found in project, manifest or registry)
 * Scallops (not found in project, manifest or registry)
 * VideoPlayer (not found in project, manifest or registry)
Please specify by known `name=uuid`.
Stacktrace:
 [1] pkgerror(::String) at /home/yakir/julia/usr/share/julia/stdlib/v1.3/Pkg/src/Types.jl:113
 [2] #ensure_resolved#101(::Bool, ::typeof(Pkg.Types.ensure_resolved), ::Pkg.Types.EnvCache, ::Array{Pkg.Types.PackageSpec,1}) at /home/yakir/julia/usr/share/julia/stdlib/v1.3/Pkg/src/Types.jl:936
 [3] #ensure_resolved at ./none:0 [inlined]
 [4] #add#25(::Bool, ::Pkg.BinaryPlatforms.Linux, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(Pkg.API.add), ::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /home/yakir/julia/usr/share/julia/stdlib/v1.3/Pkg/src/API.jl:97
 [5] add(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /home/yakir/julia/usr/share/julia/stdlib/v1.3/Pkg/src/API.jl:72
 [6] #add#24 at /home/yakir/julia/usr/share/julia/stdlib/v1.3/Pkg/src/API.jl:69 [inlined]
 [7] add at /home/yakir/julia/usr/share/julia/stdlib/v1.3/Pkg/src/API.jl:69 [inlined]
 [8] (::JLSO.var"#20#24"{Dict{String,VersionNumber}})(::String) at /home/yakir/.julia/packages/JLSO/MIDXJ/src/deprecated.jl:170
 [9] #mktempdir#18(::String, ::typeof(mktempdir), ::JLSO.var"#20#24"{Dict{String,VersionNumber}}, ::String) at ./file.jl:634
 [10] mktempdir at ./file.jl:632 [inlined] (repeats 2 times)
 [11] _upgrade_env(::Dict{String,VersionNumber}) at /home/yakir/.julia/packages/JLSO/MIDXJ/src/deprecated.jl:154
 [12] upgrade_jlso(::Dict{String,Dict{String,V} where V}, ::Val{2}) at /home/yakir/.julia/packages/JLSO/MIDXJ/src/deprecated.jl:130
 [13] upgrade_jlso(::Dict{String,Dict{String,V} where V}) at /home/yakir/.julia/packages/JLSO/MIDXJ/src/deprecated.jl:90
 [14] read(::IOStream, ::Type{JLSOFile}) at /home/yakir/.julia/packages/JLSO/MIDXJ/src/file_io.jl:40
 [15] load(::IOStream) at /home/yakir/.julia/packages/JLSO/MIDXJ/src/file_io.jl:89
 [16] #37 at /home/yakir/.julia/packages/JLSO/MIDXJ/src/file_io.jl:87 [inlined]
 [17] #open#271(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(open), ::JLSO.var"#37#38"{Tuple{}}, ::String) at ./io.jl:298
 [18] open at ./io.jl:296 [inlined]
 [19] load(::String) at /home/yakir/.julia/packages/JLSO/MIDXJ/src/file_io.jl:87
 [20] top-level scope at REPL[2]:1

[oxinabox]

I think the correct behavour is if anythjing goes wrong during upgrade, we print a warning,
then just make the Manifest and Project empty or something.

[rofinn]

During an upgrade, if we can't add specific package versions then we fallback to just trying to the package without any version constraints, since they're sufficiently different between julia 0.6 and 1.0. I don't think having an empty manifest or project is the right approach.

Overall, I don't think it's JLSO's responsibility to figure out which packages are needed for the data being saved. I'll note that in this situation there is enough information that you can simply reconstruct the file by simply modifying the BSON data directly, which is the primary benefit of the JLSO format.

[oxinabox]

I don't think having an empty manifest or project is the right approach.

I disagree.
If all else fails it is still better to be able to load the data than not.

[rofinn]

You can still load the raw dictionary with BSON though, the point of JLSO is to avoid throwing away information about the original environment as best as possible.

[oxinabox]

We can always display a warning and a recommendation to pin JLSO to old version til we fix what ever bug.

[oxinabox]

@rofinn said:

@yakir12 If you're positive that you don't need those packages then you could just modify the bson and rewrite the file.

using BSON
d = BSON.load("trackdata.jlso")
for pkg in ["Bills", "Budgets", ...]
    delete!(d["metadata"]["pkgs"], pkg)
end
BSON.bson("trackdata-reduced.jlso", d)

Then you can load the new file as you'd expect.

[rofinn]

We can always display a warning and a recommendation to pin JLSO to old version til we fix what ever bug.

The issue is that this isn't a bug... this is exactly what JLSO was designed to do. In this case, the issue is that the old environment cannot be constructed with the information available and we need user intervention to tell us what data we can throw away during the upgrade. There is the JLSO.upgrade(src, dest, project, manifest) method which we could mention in the error message, but that should be something that users explicitly call rather than us falling back to throwing away the pkg information.

[yakir12]

So I just went crazy and blindly deleted all the packages JLSO complained about, realizing that (actually) some of those are needed... But, it all just worked! It took a bit of time, installed a bunch of things it needed, but everything works. Very happy.

In the future, it might be prudent to warn the user, when saving a jlso file, that it's always better for posterity if all the used packages are online if not registered and that local packages should be moved online (and add TheOnlineVersion).

The trick of knowing which packages are needed and which are not (when users can just write using PackageThatIsNotUsed) is I image difficult, but it would be nice. I often add and using packages I end up not needing when I develop stuff and need to clean up later. A tool for that would be nifty.