loading Gadfly takes a long time

[vks]

Just loading it takes about 30 s on my machine. Is this normal? Why does it take so much time?

> time julia -e 'using Gadfly'
Warning: Possible conflict in library symbol dtrtri_
Warning: Possible conflict in library symbol dgetri_
Warning: Possible conflict in library symbol dgetrf_
Warning: could not import Base.foldl into NumericExtensions
Warning: could not import Base.foldr into NumericExtensions
Warning: could not import Base.sum! into NumericExtensions
Warning: could not import Base.maximum! into NumericExtensions
Warning: could not import Base.minimum! into NumericExtensions
Warning: could not import StatsBase.bandwidth into Stat
Warning: could not import StatsBase.kde into Stat
       29.34 real        29.03 user         0.27 sys

[dcjones]

That's pretty normal, unfortunately.

The issue is simply that using Gadfly loads a lot of Julia code. It's one of the larger Julia packages, plus it also loads several other large Julia packages like DataFrames, Datetime, and Distributions.

It's frustrating, but I don't think it's a permanent situation. A lot of work has already been done on pre-compling Julia code, and when that's extended to packages, things should be much better.

Until then, there's not much I can do about it besides removing features from Gadfly, which I'm not inclined to do.

I think relevant Julia issue you may want to watch is JuliaLang/julia#4373.

[timholy]

You can already precompile packages. Just create a userimg.jl file in base/, with stuff like

Base.require("Gadfly")
Base.require("SomeOtherPackage")

in it. Then build julia. You should find that saying using Gadfly gets just a wee bit faster.

For packages that require some kind of initialization (most commonly, those that make use of C libraries), sometimes it's more involved, and for a couple of them I've never succeeded. You can see some examples of initialization functions in Gtk, HDF5, and Images.

I've found, however, that I'm generally unlikely to precompile packages that I work on myself a great deal, because it means any change forces you to rebuild Julia. I'm still feeling my way forward on how to choose which packages to precompile and which packages to just pay the price of load time.

[vks]

Do you know whether Julia will at some point support automatic compilation as in Python?
At the moment, it seems extremely inefficient to recompile every package when loading.

[StefanKarpinski]

Making this stuff more convenient is definitely desirable and will happen.

[timholy]

@vks, since I've hardly used Python, I'm not sure I can answer your question, but I presume what you mean is "independent files precompiled for each package." I can't say when that will happen.

it seems extremely inefficient to recompile every package when loading

Agreed. But do keep in mind that the procedure I outlined allows you to avoid precisely this problem, with very little effort (in the cases where it works). One extra tip: there are actually two steps, parsing/lowering (which is mostly what happens when you load a file) and JIT-compiling. The Base.require() trick will accomplish the former; you can force the latter for particular functions & types using precompile (which you can read about with ?precompile). See an example package implementation in Images.jl/precompile(); you'd need to call this in your userimg.jl directly.

[vks]

@timholy: In Python, whenever you load a module, the corresponding source file (.py) gets compiled to bytecode and cached in the same directory (.pyc).

The nice thing is that the user does not need to care about it, it just happens.

I'm not sure how Julia works internally, but it would be great if it could save time doing something similar.

[timholy]

That would be nice. It will happen when someone makes it happen. (We're all volunteers here.)

[dchudz]

"I've found, however, that I'm generally unlikely to precompile packages
that I work on myself a great deal, because it means any change forces you
to rebuild Julia." <== Does it work to "reload" a precompiled package if
want to incorporate changes and don't want to rebuild Julia?

(Thanks for the precompiling tip! That's nice to know about.)

On Thu, Mar 27, 2014 at 3:31 AM, Tim Holy [email protected] wrote:

That would be nice. It will happen when someone makes it happen. (We're
all volunteers here.)

Reply to this email directly or view it on GitHubhttps://github.com//issues/251#issuecomment-38787644
.

David J. Chudzicki
blog.davidchudzicki.com
[email protected]
(518) 366-7303

Data Scientist
Kaggle (we're hiring!)

[timholy]

Good question, I'm not sure I tested, but I bet it would. Usual rules apply, of course---you'd have to scope everything with the module name for the new code to be used.

[ViralBShah]

FWIW, @tanmaykm does include Gadfly in the sys.ji on JuliaBox.

[tanmaykm]

Yes, Gadfly and its related packages are precompiled on JuliaBox and that makes it load almost instantaneously. Also, reload of the package works as @timholy mentioned.

[stevengj]

If you add VERSION >= v"0.4.0-dev+6521" && __precompile__() to the top of Gadfly.jl it will now (in the latest Julia 0.4-dev) precompile automatically the first time it is imported, and recompile automatically thereafter when any of the dependencies are updated.