Update Travis CI build to try building docs

[jiahao]

This PR adds automatic doctest builds to Travis. By including the doctests as part of the Travis test suite, the hope is to prevent the documentation examples from becoming so far out of sync with reality as to require serious manual updating, as in #7693.

This is the brute force solution to #7691.

Four documentation tests are included here:

• HTML documentation build test
• PDF documentation build test (which can catch some embedded HTMLisms)
• Doctest build test (runs example blocks and compares them with the embedded output)
• Link test (checks that links are still current)

[ivarne]

Does it make sense to test documentation both in GCC and CLANG?

[jiahao]

In theory, the binaries built by gcc and clang could produce different output (e.g. memory addresses in printouts). If build-dependent information exists in the documentation, that should certainly be checked for.

[JeffBezanson]

What can we do about all the random numbers?

[ivarne]

Hidden srand(0)?

Or just have them in a code block instead of a doctest block.

[jiahao]

If you meant tests using rand/randn, the docs are set up with .. testsetup:: blocks that fix the random seeds. (At least one test had to be emended in #7693 to be reproducible.)

If you meant things like line numbers and method counts, I don't have any good ideas.

[JeffBezanson]

Of course we want to err on the side of testing more things automatically, but we'll have to see how annoying the changing line numbers are in practice. I'm afraid it might be very annoying.

[jiahao]

Actually I have an idea about the line numbers. Since we use our own Python classes anyway to run the tests in JuliaDoc, it is possible to intercept the test check here with a custom SphinxDocTestRunner._checker that will mod out differences in line numbers, memory addresses, etc.

[jiahao]

On an unrelated note, it also occurs to me that JuliaDoc itself could be a good candidate for using pyjulia.

[jakebolewski]

That would be conditional on pyjulia actually working :-(. I haven't looked at it in months and it is probably broken, although one intrepid windows user has gotten it to work for him.

[jiahao]

I've rebased and updated this PR so that only the basic HTML and PDF docbuilds are run.

I realize that these tests add more time to the Travis build, but it takes only about a minute or two to run and will catch syntax errors in changes to the documentation, which are unfortunately quite frequent since not many people are familiar with the nuances of ReStructuredText.

As for the doctests, I think running them as part of the release-candidate target is the best we can hope for at this point without an extensive rewrite of JuliaDoc.

cc @tkelman @staticfloat

[tkelman]

Related: #9553 #9554
We shouldn't need to touch apt-get, pip, or brew upgrade gcc here, I think.

[jiahao]

I'll remove the gcc line, it must have been left over from the rebase.

The pip lines are necessary to install sphinx and the doc theme correctly.

[jiahao]

Oh, the new virtualenv setup means that it's no longer necessary to install sphinx manually. Let me try that.

[staticfloat]

Alternative idea; We could add an entry to the build environment matrix that is a "no-arch" run, that just builds the docs and not Julia. It can then run in parallel to the other jobs, and won't run multiple times.

[tkelman]

That's exactly the approach I took in #9553, which I am increasingly convinced no one other than me has ever looked at.

[jiahao]

Sounds good. I think @tkelman proposed something similar in one of the other issues.

This afternoon I played around with trying to make Travis short-circuit the build if only doc files were changed. (see the cjh/probe-travis branch)

I've figured out what to do as a git pre-push hook here, but Travis appears to choke on the ;; in the case statement and I haven't figured out how to fix it. Perhaps you'll have some idea.

[tkelman]

I don't know where you're getting the inputs from, but note that $remote_sha from Travis is generally not accurate. Travis typically sets it to the sha of a merge commit at time of build queuing, not the actual merge commit that gets fetched.

Travis appears to choke on the ;; in the case statement and I haven't figured out how to fix it

If you try to write too much bash in yaml you will only end up tearing your hair out. Write a script under contrib or contrib/ci and just call it.

[jiahao]

The pre-push hook is for my local use with git; it will have to be adapted to the Travis environment.

note that $remote_sha from Travis is generally not accurate

I did notice this and tried to substitute as proxy the nearest branch point. I tried the magic incantation from this SO thread

PARENT_BRANCH=`git show-branch | grep '*' | grep -v "$(git rev-parse --abbrev-ref HEAD)" | head -n1 | sed 's/.*\[\(.*\)\].*/\1/' | sed 's/[\^~].*//'`

Knowing the parent branch allows you to retrieve the commit at the branch point with

git rev-list -n 1 $PARENT_BRANCH

This works locally for me, in that branches off master gave PARENT_BRANCH=master and similarly for release-0.3, but not on Travis

https://travis-ci.org/JuliaLang/julia/builds/49214143#L136

I think owing to the clone being shallow.

[tkelman]

There's also something a little strange about the way remotes work on Travis that can make version_git.sh behave oddly.

[jiahao]

I'll leave my git experiments here for now. It's time to ford the 3' snowdrifts in search of suds and vittles.

[tkelman]

No longer relevant, we've been building the docs in make install for some time.

[jiahao]

But not the manual or on Travis still.

[tkelman]

Good point, guess we could reopen and/or try again.