Install and test embedding examples

[tkelman]

I'm starting to think we should install examples/embedding.c and examples/Makefile along with examples/*.jl and test on Travis that it keeps working in an installed version of Julia. For this to make sense, the makefile would need to be modified quite a bit (or a new one autogenerated?) to be standalone.

Related to #8428, cc @nalimilan

[waTeim]

Ok seems like I'm the guy to do this, as I'm likely the one that would benefit most from it. Is there any sort of introductory information, etc?

[tkelman]

Our current make install is pretty much roll-our-own in the top-level makefile. Step one is make sure the current embedding example and Makefile are working properly. If that's all fine, then it's a matter of trying to decouple examples/Makefile from Make.inc to make it usable in a non-source build of Julia. There are various flags and paths that we could probably just cat into the start of the installed copy of examples/Makefile.

I've seen this done really well in a few autotools projects, but I think for this isolated case we can do something simpler without needing to redo the rest of the build system. If you happen to have done Pkg.add("Ipopt") on a Linux machine I can point you to what Ipopt's example makefiles look like, which I think are a good model for this.

[waTeim]

The critical question on OS/X and Linux is where are the header files (for -I) and where is libjulia.so (for -L and -rpath). Is that all occurring assuming it's been deployed (like in /usr/local?), or is the directory known? On Windows, I don't know. Is this something that will be part of Travis and how does it get invoked? It would be pretty easy to pull together an autotools config that will look for Julia in a bunch of standard places, but problem with autotools is on Windows, doesn't that imply cygwin?

[tkelman]

So, the top-level makefile that does make install will know where everything's going, and can put that information into the installed version of the example's Makefile. We can add some lines to .travis.yml to cd into the installed copy of the examples folder, run make, and verify the output. I wasn't going for "looking for Julia in a bunch of standard places," since it'll only ever be in a standard place if it's packaged by a distribution. It would be nice to allow this to work for those distributions, so people who get Julia from a distro package can have an installed example embedding Makefile that will have the correct paths for their system. Our binary installs that aren't from distro packages, either the Linux tarball or the OSX/Windows installers, may end up at any location. So hardcoding absolute paths probably isn't a good idea, but if we can make everything relative-path based and self-contained I think it could be made to work.

[waTeim]

If autotools, there there would eventually be some variable INSTALLDIR which could be overridden on the command line to point to the exact path during compilation, otherwise it searches, autotools style. For reference, here's an approach to do that definitely not autotools style, but gives an idea about where to look. Maybe some simplified version of this in a bash-autotools way?

[tkelman]

We have a bunch of pseudo-autotools-ish variables like that in our makefiles already. For install-time logic it probably makes sense to do the file modifications as part of the install target in the top-level Julia makefile.

Instead of doing any searching through system paths, I think we can set a default (relative?) path based on the normal layout of an install. Ideally set up in such a way that if distributions customize the installation layout, that would get reflected in the default path we put in the embedding makefile.

[waTeim]

I think there are at least a couple distros like that. pg_config and pythonX.Y-config come to mind. How about in addition to the julia executable there's a program called julia-config? There are a few holes, though, what if Julia is installed multiple times in different locations (simultaneously 0.3 and 0.4) then what? Actually, looks like the python-guys thought of that. How to find julia-config? Maybe assume the user will install in such a way so that the Julia he wants to use will have the corresponding julia-config first in PATH.

[tkelman]

That might not be a bad idea, llvm and pcre have the same sort of thing. There's also the classic, but marginally difficult-to-use-on-Windows, option of shipping a julia.pc pkg-config file. Related: #8650

[tkelman]

I'm wondering out loud here if, aside from any modifications to the embedding example, the more complicated aspects of this can be done out in a central package, PkgConfig.jl or EmbeddedConfig.jl or whatever, to collect convenient scripts, code snippets, etc that would be useful across multiple targets for embedding Julia - you with your node-julia work, pyjulia, the various efforts for calling Julia from R and Matlab, etc. Right now it seems like you're all reinventing the same bits and pieces, some of which is inevitable to fit with the different build systems each of those environments use, but maybe cross-target collaboration could save everyone some work in the long run. If you're calling Julia from some other language, it's probably safe to say you have Julia installed, right?

[waTeim]

Yea right now we're all a bunch of kangaroo's jumping around. We'd all jump to a common thing if it fit our needs.

I'm wary of something like EmbeddingConfig/PkgConfig.jl unless it was guaranteed part of base or Pkg.update() is not required as It needs to respond quickly and never fail. Putting stuff in a Pkg allows for the flexibility of the meta information contained by the package to change when the Julia install does not change. Would that be useful?

[tkelman]

I guess the argument elsewhere was "if it needs to be kept up-to-date with changes to base, it should be in base" but I'm also not sure how often these things will change. The slowness of Pkg.update() is annoying but will hopefully get better once the libgit2 transition is done? An alternative would be to make it a script or set of scripts in contrib (that get installed with binaries) the way build_sysimg.jl was just done.

[ViralBShah]

Should we target this for 0.3.4, or is it too close and go for 0.3.5?

[tkelman]

I think this is an 0.4 target, not an 0.3.x one. It can probably be backported when it's ready, but there isn't even a PR for it yet and 0.3.4 is scheduled for a few days from now...

[waTeim]

I'll try something and we can at least talk about something concrete, but yea, I don't see it happening in a few days.

[ViralBShah]

Thanks @tkelman @waTeim

[waTeim]

Can you point me to a list of the pseudo helper tools so I can kind of get jump started? I'm going to start with the approach of having an auxillary program called julia-config like above, and it would be good if it were written in Julia if possible. Would such a thing be fast enough?

[staticfloat]

You're going to be looking at Make.inc a lot. These variables are a good starting point, where prefix is the root directory where Julia will be installed (for instance, on my Ubuntu ppa, prefix gets set to /usr/local).

[tkelman]

What do you mean by "pseudo helper tools"? As with the build_sysimg.jl case, I'd prefer Julia scripts keep the extension for the sake of Windows users. We can merge whatever you come up with even if it doesn't completely cover Windows as a first step, but naming-wise, shebangs don't work in general cross-platform.

[tkelman]

You'll also likely want to look at the already-autogenerated file build_h.jl, it has some handy paths-relative-to-JULIA_HOME defined, if you need to add a few more things it might be convenient to do so there.

Though I think all of those end up going into Base already so maybe nevermind... but it is an example of Makefile-generated Julia code similar to what you probably want to go for.

[waTeim]

Whoops, I meant the "pseudo-autotools-ish variables" you mentioned above.

[tkelman]

okay, yeah, see the lines @staticfloat linked to (

julia/Make.inc

Lines 93 to 104 in 3109867

	# Directories where said libraries get installed to
	prefix ?= $(abspath julia-$(JULIA_COMMIT))
	bindir = $(prefix)/bin
	libdir = $(prefix)/lib
	private_libdir = $(libdir)/julia
	libexecdir = $(prefix)/libexec
	datarootdir = $(prefix)/share
	docdir = $(datarootdir)/doc/julia
	mandir = $(datarootdir)/man
	man1dir = $(mandir)/man1
	includedir = $(prefix)/include
	sysconfdir = $(prefix)/etc

for readers from the future) - also the relative paths a little further down

[staticfloat]

Darn, forgot to canonicalize! Tony catches so many of my mistakes, it's rather embarrassing. :P

[tkelman]

Linus' law. Our eyeballs just happen to be frequently pointing at the same bits of code.

[tknopp]

+1 installing the embedding example.

But for Travis testing wouldn't it suffice to build and execute the example in a source build? This would have to be invoked by the regular unit test framework.

[nalimilan]

Sorry, I haven't really followed this conversation, but my 2c is that I'd prefer tests to be only in $datarootdir/julia/test/, and not in $docdir/julia/examples, as I'm putting the latter in a separate -doc subpackage.

[tkelman]

But for Travis testing wouldn't it suffice to build and execute the example in a source build?

Partially, not completely. Testing the embedding example in a source build only makes sure we don't break it in a source build. Testing it from the installed copy makes sure it works in an install, which would be more valuable.

[waTeim]

Is #9438 useful here as a means of creating a julia-config?

[tkelman]

Don't think so. That approach is temporary anyway. If you are operating under the assumption that Julia has to be installed in order to build an application that embeds it, then you don't need to go to any lengths to make a julia-config script usable in a standalone way. Just install it to joinpath(JULIA_HOME, Base.DATAROOTDIR, "julia") and ensure the user provides the right path to the specific julia executable they want to use.

[waTeim]

I copied the embedding.c file from examples into a test directory and modified embedding.c and Makefile to follow the pattern in #10069:

One change to embedding.c:

jl_init(NULL);

changed to

jl_init(JULIA_INIT_DIR);

and then the makefile changed to

JL_SHARE = $(shell julia -e 'print(joinpath(JULIA_HOME,Base.DATAROOTDIR,"julia"))')
CFLAGS   += $(shell $(JL_SHARE)/julia-config.jl --cflags)
CXXFLAGS += $(shell $(JL_SHARE)/julia-config.jl --cflags)
LDFLAGS  += $(shell $(JL_SHARE)/julia-config.jl --ldflags)
LDLIBS   += $(shell $(JL_SHARE)/julia-config.jl --ldlibs)

all: embedding

followed by make, then running it....

bizarro% make
cc -DJULIA_INIT_DIR=\"/usr/local/julia-f875a541be/lib\" -I/usr/local/julia-f875a541be/include/julia  -L/usr/local/julia-f875a541be/lib/julia  embedding.c  -Wl,-rpath,/usr/local/julia-f875a541be/lib/julia -ljulia -o embedding
bizarro% ./embedding 
1.4142135623730951
sqrt(2.0) in C: 1.414214e+00
sqrt(2.0) in C: 1.414214e+00
x = [9.000000e+00 8.000000e+00 7.000000e+00 6.000000e+00 5.000000e+00 4.000000e+00 3.000000e+00 2.000000e+00 1.000000e+00 0.000000e+00 ]
my_func(5.0) = 10.000000
1.4142135623730951
UndefVarError(:this_function_does_not_exist)

[tkelman]

cool, that's the expected output, right?

[waTeim]

Ya, pretty sure. I can check it. But, is there a thing that does it automatically? How much do I need to modify, and how much to add? Is there a thing that installs Julia in test yet for instance?

[tkelman]

We install julia on Travis. That code is getting a bit messy but we can fiddle with it.

It would be ideal to make examples/Makefile not rely on include $(JULIAHOME)/Make.inc any more, so it can work in a binary install. I'm not sure yet how hard this will be.

Also the path to the julia binary may as well be set in a variable so it can easily be overridden

[waTeim]

I didn't look ultra deeply, but it appeared that all Makefile does is build embedding, if so then modifying like above should do it, so long as Julia is in PATH, everything else is automatic. Is the examples directory installed somewhere? Does make test do this?

[tkelman]

so long as Julia is in PATH

Or the location is easy to specify by overriding a JULIA variable on the command-line.

Is the examples directory installed somewhere?

I believe make install does install some but not all of the examples directory. Some of it was removed in #8428 which you might now want to put back if it can be made to work in a binary install.

make test

There is a test/examples.jl that makes sure the code there still runs. I think this might be better to do as a new file test/embedding.jl, since it will depend on having a toolchain installed so we should make it easy to skip.

[waTeim]

Ok I know now this must be completed. There is no check for changes that break embeddings, and it happens a lot.

To fix this, I need some help in understanding how the current test suite functions and how a new embed test would need to work (HELP).

As I said (months ago, heh) this should be pretty easy, but currently I am not familiar at all with this test, what occurs, how it runs.

A good start would be can someone tell me what to run the test locally? Like "make test" or something?

[tkelman]

Yeah, start with make test from a source build. If you add a new file test/embedding.jl and add "embedding" to the list in test/choosetests.jl then the content of test/embedding.jl will become part of the test suite, and get run by Travis, AppVeyor, etc. For binary builds, it's also ideal if Base.runtests("all") passes, but I'm not sure how often that gets checked so it might break once in a while.

[waTeim]

This would be different. This would need to make and run an external test separated from all of that seems like. Maybe what could be done is in embedding.jl there could be some invocation of external commands: make embed1,embed2, etc. It would also need to occur against an "installed" version of julia as opposed to within the source tree if it's going to be accurate. I'm not sure that the current test framework imagines that something like that could happen.

[tkelman]

Maybe not. Travis at least does do make install, but assuming make is present isn't always a safe assumption to make for Windows, or OS X when the command-line tools aren't installed. It is safe to do that kind of thing on Travis though.

Maybe another option would be to use the framework of a Julia package to set up this testing? With PackageEvaluator and Travis at least it will run pretty regularly and in a mostly-automated way. Could have an EmbeddingExamples.jl package with code snippets, build scripts, etc? Not sure if that's better or worse than having things live in the base repository but not tested very often.

[tkelman]

@twadleigh sounds like you're about to do at least the test part of this? xref #11419 and #18265

[twadleigh]

@tkelman thanks for the reminder of this thread. Yes, I'll at least do the test part. I'll look into the install part while I'm at it. Since we have this issue, I think we can close #11419.

[tkelman]

Mostly fixed by #21299, subject to some rpath caveats. We're not putting the built embedding executable in binaries just yet, but the rest of this is done now.