0.3.6 release planning issue

[tkelman]

The last release on the release-0.3 branch was tagged 26 days ago, so it's getting to be about that time to get the outstanding backports in and tag a new one over the next week or two. Opening this to centralize discussion.

I don't think we have any pending PR's against release-0.3 right this second, correct me if I'm wrong though. If there are currently open or recently merged PR's, or recent commits straight to master, that should be backported but are not yet labeled, please go through and flag them. Add a mention of @JuliaBackports if it is a commit that did not come from a PR (or if you are not a Julia collaborator and don't have permissions to edit issue labels), otherwise I prefer using the label for tracking wherever possible.

[tkelman]

Linking to the backport pending list: https://github.com/JuliaLang/julia/issues?q=label%3A%22backport+pending%22+

I think the only questionable one is the fix to #9544 - the signal handling changes there seemed to introduce a fair amount of CI instability on master. Or they happened to be committed around the same time as some other underlying cause. Not sure. Getting others' input about whether or not to backport 40c2225 and the followup fixes would be good.

I would get to backporting the uncontroversial ones now, though our appveyor queue is currently really long so it might be nice to wait until a quieter time of day.

[kmsquire]

Just to double check: I just added a the backport pending label to a couple of issues (#9936, #9952), as well as marking them with @juliabackports. Is that the correct procedure, or should I have just used one or the other?

[tkelman]

Where there's an issue, I personally think the @juliabackports mention is a bit redundant. It helps to keep the noise down on that mailing list (https://groups.google.com/forum/#!forum/julia-backports) to only things we can't track any other way.

And the milestone doesn't really hurt either if you want to add that, but I also don't think it's strictly necessary.

[kmsquire]

Okay, thanks!

[IainNZ]

Related in the vaguest sense: I just added lines to the chart at http://pkg.julialang.org/pulse.html to show releases.

[ViralBShah]

@tanmaykm It would be nice to bring the sparse matrix indexing updates into 0.3.6, which fix the bugs and performance issues too. Unfortunately, I don't think that cherry picking will work.

[tkelman]

@ViralBShah bugfixes sure, but I'd rather be conservative on feature/performance changes. Prepare a PR soon and we can decide.

[ViralBShah]

This may be a bit involved, so let it not hold up the release. I may not get to it for another week or so.

[tkelman]

If possible it would be good to fix #10085, an apparent regression caused by backporting the fix for #8551.

[vtjnash]

I think the only questionable one is the fix to #9544 - the signal handling changes there seemed to introduce a fair amount of CI instability on master. Or they happened to be committed around the same time as some other underlying cause. Not sure. Getting others' input about whether or not to backport 40c2225 and the followup fixes would be good.

that was not caused by the raise test (or related changes), but by the addition of the invoke test just before it

i agree that the fix for #10085 should be backported (and perhaps the rest of the related fixes to #8551, although they are mostly unrelated to #10085)

[tkelman]

Okay, were there any other related followup fixes on signal handling that I'm forgetting, or should 40c2225 squashed with 84464eb and 1f11824 be sufficient and stable enough for release-0.3?

[vtjnash]

i thought there might have been another one, but maybe the remaining commits were just turning that test on and off

[tkelman]

We're getting past due on this. I can't figure out how to resolve any of the conflicts in anything else that's been marked for backporting. Either someone else will have to do it (preferably open PR's against release-0.3 so things can run through CI), or we can just tag as-is.

[nalimilan]

I would be nice if #10201 could be fixed so that 0.3.6 can be shipped in Fedora rawhide.

[tkelman]

Sure, worth bisecting to see if that's a regression on release-0.3 caused by some other backport.

[staticfloat]

Yes, thanks for the reminder @tkelman. I'm in favor of tagging as-is as soon as things are able to compile on Fedora. @nalimilan can you do the bisect, or should I?

[tkelman]

It looks like we needed to backport 779bf96 otherwise source builds of release-0.3 with LLVM 3.5 releases wouldn't work correctly. I got a successful build after that backport on Ubuntu 14.04 with LLVM 3.5.1, I'm trying 3.5.0 right now to see if things are any different. If not, it's some difference caused by USE_SYSTEM_LLVM and might be trickier to debug.

[tkelman]

Also #10200 (cc @jiahao) is open which is relevant for using Intel compilers but I'd really prefer that to be tested on master for several days before merging into release-0.3.

[tkelman]

We've got a little problem and a big problem. The little problem is this doctest: https://github.com/JuliaLang/julia/blame/723280cea12991b7370b18f274ded993e9275e34/doc/manual/metaprogramming.rst#L288 is failing due to some doctest scope issue, a is apparently defined (from an earlier doctest?) but b isn't when neither should be. fixed in 50ea614

Big problem is julia-debug.exe does not start, it just segfaults and dies. I'll need to bisect this one.

[tkelman]

The julia-debug.exe thing might not be a blocker. 0.3.3 is ok, 0.3.4 is bad, but removing sys.dll seems to fix the issue, and we don't distribute sys.dll yet. There's some very strange interaction going on between julia-debug.exe and sys.dll on both win32 and win64. According to @vtjnash in #10111 julia-debug should not be loading sys.dll at all. Not sure if the situation is any different there between master and release-0.3

[nalimilan]

Forget about #10201 for 0.3.6, looks like it's a bug in Fedora/gcc 5.0.

[ViralBShah]

We could backport #10206 cleanly, I am guessing, for significantly faster sparse vcat performance. I am not sure if we should, but just noting it here to see what others think.

[ivarne]

The fix for #8631 is also tempting, but that one needs more time to catch potential issues on master.

[IainNZ]

Surely at this point we should be shifting towards a purely bug-fix strategy for 0.3, not performance improvements, and being extremely risk averse?

[StefanKarpinski]

I'm with @IainNZ on this. Performance enhancements are not worth it for 0.3.

[ViralBShah]

I think that is the right thing to do. We have done that in the past, which is why I just brought it up.

[tkelman]

Agreed on being conservative, especially for 0.3.6. The last 2 tags were pretty close together, but it's been 39 or so days since 0.3.5.

We have a few outstanding issues, but nothing that wasn't also a problem in 0.3.5. No one has responded to my requests for assistance in resolving conflicts on the fixes that have been flagged for backporting, so my vote would be to merge #10200 then tag.

[staticfloat]

I've merged #10200, I'm going to make release-candidate and build binaries for people to test. If those binaries look good, I'll tag and then rebuild final binaries. That process will give us a window to make sure the binaries are good before tagging so we don't have to move tags around like we have done in ages past.

[nalimilan]

The build passes on Fedora/RHEL, so it's OK for me.

[staticfloat]

Binaries are available for testing:

OSX (tested on my laptop to be working)
Linux x64 (tested on my 64-bit desktop to be working)
Linux x86 (tested on my 64-bit desktop to be working)

Windows x64
Windows x86

[tkelman]

gtg from me on win32 and win64

[tkelman]

We seem to have lost "The Julia Programming Language" process description that shows up in task manager some time between 0.3.3 and now...

[pao]

It's still present in 0.3.4.

[tkelman]

Strangely it works fine when I build Julia locally, even latest release-0.3 branch. Is this something having to do with signing? Or are the Windows buildbots being kept up-to-date on the toolchains from Cygwin?

[tkelman]

Nevermind, it turns out that just happens if you run the installer to one location then rename the folder. Windows is odd. Carry on.

[pao]

Narrowing it back down, I'm using an official 0.3.4 Windows x86-64 build.

[staticfloat]

Great, this is enough testing for me. I'm going to tag it and build 0.3.6 binaries. @nalimilan feel free to do your thing.

[nalimilan]

Damn, I see this on both RHEL6 and Fedora 21, 32 and 64 bits (other builds have not completed yet):

/builddir/build/BUILD/julia-0.3.6/build/usr/bin/../lib/julia/sys.ji
/builddir/build/BUILD/julia-0.3.6/build/usr/bin/julia --check-bounds=yes -f ./runtests.jl spawn
     * spawn
       [stdio passthrough ok]
exception on 1: ERROR: test failed: readall(@cmd "\$exename -f -e 'println(STDERR,\"Hello World\")'" .> @cmd "cat") == "Hello World\n"
while loading spawn.jl, in expression starting on line 154
ERROR: test failed: readall(@cmd "\$exename -f -e 'println(STDERR,\"Hello World\")'" .> @cmd "cat") == "Hello World\n"
while loading spawn.jl, in expression starting on line 154
while loading /builddir/build/BUILD/julia-0.3.6/test/runtests.jl, in expression starting on line 39
/builddir/build/BUILD/julia-0.3.6/build/usr/bin/../lib/julia/sys.ji
make: *** [spawn] Error 1

https://copr-be.cloud.fedoraproject.org/results/nalimilan/julia/fedora-20-x86_64/julia-0.3.6-1%2bcopr.fc21/build.log
https://copr-be.cloud.fedoraproject.org/results/nalimilan/julia/epel-6-i386/julia-0.3.6-1%2bcopr.fc21/build.log

I don't understand how that's possible, given that the build succeeded yesterday...

[staticfloat]

I goofed and had a bad tag that had a single debugging line slip through in the actual tag commit. The correct tag sha is 0c24dca, can you verify that's what you have?

[nalimilan]

Indeed, the checksum of the tarball has changed. I've started a new build: https://copr.fedoraproject.org/coprs/nalimilan/julia/build/78177/

I need to go to bed, but if it succeeds (disregarding on rawhide where it will fail because of LLVM), it's OK. (I'll have to sort out how to disable downloading sphinx to make the official Fedora package, but it can wait a bit more.)

[staticfloat]

Looks like everything is working just fine. The juliareleases PPA isn't ready yet, (got some packaging problems with virtualenv right now; @nalimilan if you can point me to the relevant part in your .spec file where you stop virtualenv from trying to download sphinx, that'd be great) but other than that, the release is good to go!

Good work everybody, and a special shout out to @tkelman who really helped push this out the door as I (and I assume many others) have been super busy with non-Julia related things.

[nalimilan]

Good. Actually, as I wrote above, I also get the failure due to virtualenv. Glad to know you've got the same problem! ;-) Maybe the easiest solution is to take the git commit and make a patch reverting it. I used to apply this patch in 0.3.5 and it worked: http://pkgs.fedoraproject.org/cgit/julia.git/tree/julia_juliadoc.patch

[tkelman]

Thanks @staticfloat for hitting the buttons on the buildbots, and @nalimilan for testing and handling fedora-and-friends.

Now that we've just tagged, it would be the perfect time for people to help with resolving conflicts and prepare backport PR's against release-0.3 for #9544, #10085, and/or #8631.

[nalimilan]

@staticfloat I've updated my patch to build the docs without virtualenv. I only had to change a few lines to the existing patch (added the two last files):
http://pkgs.fedoraproject.org/cgit/julia.git/tree/julia_juliadoc.patch?h=f22&id=f582353588d679a2596af7cdbc14cd91f5f3c1d3

For 0.4 we really need to build the docs in the tarball (#8378).

[nalimilan]

BTW, @sebastien-villemot, the patch may be useful to you too.