Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

saving files #111

Closed
behinger opened this issue Apr 25, 2023 · 1 comment · Fixed by #114
Closed

saving files #111

behinger opened this issue Apr 25, 2023 · 1 comment · Fixed by #114

Comments

@behinger
Copy link
Member

Saved files can be quite large - play around with @save filename.JLD2 {compress} ??

Check why files of a single subject with 128 channels can be 2gb large - seems too big

@behinger
Copy link
Member Author

function save(file,uf::T;compress=false) where T<:UnfoldModel
	jldopen(file, "w"; compress = compress) do f
    	f["uf"] = T(uf.design,Unfold.DesignMatrix(designmatrix(uf).formulas,missing,designmatrix(uf).events),uf.modelfit)
	end
end
function load(file,::Type{<:UnfoldModel})
	f = jldopen(file, "r") 
	uf = f["uf"]

	form = designmatrix(uf).formulas
	events = designmatrix(uf).events
	
	X = Unfold.modelcols(form.rhs, events)
    
	
	# reintegrate the designmatrix
	return typeof(uf)(uf.design,Unfold.DesignMatrix(form,X,events),uf.modelfit)


	
end

@jschepers can you try this out with an realistic UnfoldFile?

These tools might be helpful:

pretty_filesize(x) = Base.format_bytes(filesize(x))
pretty_summarysize(x) = Base.format_bytes(Base.summarysize(x))
	@time save(joinpath(tempdir(),"test3.jld2"),m;compress=true)
	@time save(joinpath(tempdir(),"test4.jld2"),m;compress=false)

begin
	sizex = x->pretty_filesize(joinpath(tempdir(),"test$x.jld2"))

	@show pretty_summarysize(m)
	for k = 1:4
		@show sizex(k)
	end
end

@time m2 = load(joinpath(tempdir(),"test4.jld2"),UnfoldModel)
m.designmatrix.Xs == m2.designmatrix.Xs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant