0.3.4 missing _jl_base_ctx for embedding programs

[waTeim]

Just like in #9117 for example

julia.h:

 994 #ifdef COPY_STACKS
 995 // initialize base context of root task
 996 extern jl_jmp_buf jl_base_ctx;
 997 #define JL_SET_STACK_BASE                               \

So programs that attempt to embed Julia 0.3.4 will break. What probably happened was the pull request to made stack use more efficient was back-ported, but the fix to export _jl_base_ctx was not.

[ihnorton]

@JuliaBackports

[tkelman]

Damn, this is really hard to get and keep working isn't it. We might want to ping a few of the embedder folks on the next release planning issue to check that we don't accidentally break things during "rc" backporting time. Also #8757

[waTeim]

I feel the solution to #8757 will be pretty good for things like this so with that in mind...

[vtjnash]

fwiw, this variable is gone from master (because I felt that it made initialization of julia too complicated)

[tkelman]

for the sake of the @JuliaBackports list, this was resolved in c3449a0

[waTeim]

So since this affects not only me and anyone who uses my software (with 0.3.4), but anyone in the world who is trying to embed 0.3.4, is there a way to update 0.3.4? As far as regressions, I don't see given the simplicity of the fix how it could affect anything but embedding programs, and things could only be improved.

[tkelman]

It was resolved by a dllexport in the c code, so not without recompiling. We could build unofficial binaries that are either the state of release-0.3 now (or at any time since c3449a0), or make a new branch that's 0.3.4 plus c3449a0 cherry-picked and build binaries off of that.

[staticfloat]

I am a little hesitant to make a 0.3.4.1/0.3.5 so soon after 0.3.4 was released. It would be all but trivial to build new release-0.3 binaries from the tip of the 0.3 branch for you, so if that's all you need until 0.3.5 is released at the end of January, that's easy enough.

[waTeim]

But the problem is it's not me personally. If it were just me, I'd just use 0.4; it's everyone else. I feel you about too soon, but consider every additional day is also 1 day closer to too close to 0.3.5. If it's a problem, I say fix it; I think it's a problem. In the meantime some unknown person will be trying it for the first time, get this error, shrug their shoulders and give up, and I feel that too. Waiting 1 month seems like a long time.

[tkelman]

Unfortunately embedding has not been a major first-class supported and regularly tested use case, it's a handful of people so far. It will take work to get things there so we don't accidentally break them again. We try to do a bunch of testing and make sure things are stable for the standard use cases before tagging a new release, we want it to be good enough that everyone can and should upgrade to it. Right now we're in the middle of backporting a bunch of stuff (check how many issues have the pending tag), while Travis has been very clean on release-0.3 for virtually all of these recent backports, it's a little much to make another tag 1.5 weeks after the last one to release just this (plus a handful of other already-backported) bugfix.

We could look into making say weekly snapshot binaries off release-0.3 for things like this though, that's easy enough to do without going through the whole release testing checklist.

[JeffBezanson]

It looks like there has actually been a significant number of fixes on 0.3 since 0.3.4. It might make sense to release 0.3.5 after another few days of backporting. Very quick bugfix releases are pretty common.

[tkelman]

@JeffBezanson there have been (mostly @jiahao's doc sprints, please check through that I didn't miss anything that only applies to 0.4), but there are also a bunch more pending.

[waTeim]

@tkelman Sure, I get it, I'm not super-familiar with the release process, but it's probably a lot, and don't like making busy work for you and at the same time I don't like letting those unnamed down who might use Julia for embedding and feel the need to be some sort of advocate. And of course I don't like the "build failing" tag on my own stuff. That I owe a lot to this community already is indisputable and I hope I can repay that somehow. You're thinking, well where's #8757? Yes, I owe that, I know it. If I could just put this problem behind me somehow, I can turn my attention to it. How would a weekly snapshot work/be published? 0.3.4-date? just 0.3.4 but silently replacing the previous? Would there be some sort of annotation on the differences week by week? What I'm worried about is the default user action would result in some weird discouraging error.

[tkelman]

@waTeim judging by a separate discussion (which may as well be continued publicly) it sounds like most people disagree with me and would like to do a release sooner that fixes this regression. Aside from minor questions of whether to call it 0.3.5 or 0.3.4+whatever, we need to draw a line somewhere in terms of which set of backports to include besides this one.

[staticfloat]

@waTeim I certainly don't want to give you the impression that we either don't care about correctness or want you to "just live with it/fix it yourself". We've just got a lot of concerns running in parallel with releases, and it's often difficult to push one through for a single issue without disrupting the natural flow of other fixes, enhancements and documentation that makes its way from 0.4 into 0.3.X.

As it turns out, this issue is serious enough that we've decided to just begin the release process now, figure out what can/can't make it into 0.3.5 and release it as soon as possible. I'd estimate 4 days to give us time to figure out what other fixes we're going to try and compact into 0.3.5 before distributing final binaries.

[vtjnash]

while i agree it makes sense to go through with the 0.3.5 release, I think you can also avoid this by using the julia_trampoline function instead of calling JL_SET_STACK_BASE directly. (neither will be necessary in 0.4)

[waTeim]

Perfect! I'll plan accordingly, and thanks for handling the problem with such balance and aplomb. It's great to participate in this collaboration.

[ViralBShah]

+1 for the 0.3.5 release here.

[ViralBShah]

It can be the 2015 new year special. :-)

[tkelman]

yes/no opinions for 0.3.5 on the individual backport pending issues https://github.com/JuliaLang/julia/issues?q=label%3A%22backport+pending%22+sort%3Aupdated-desc would be helpful, especially on the ones that aren't just doc (or doc-building-process, I think we'll push that one back) improvements