Fix #160 Enable optimizations

[blopker]

Gist of analysis: https://gist.github.com/blopker/9fff37e67f14143d2757f9f2172c8ea0

Fixes #160

[blopker]

The commits still need to be squashed, but the PR is ready for review now.

[JayH5]

@blopker thanks for giving this a try. Those build times are really long, though 😞.

Maybe you could make a separate PR just to do the FFI/Expat stuff for Python 2.7? It seems like a good idea to fix that and I don't think we should let the --enable-optimizations discussion get in the way of that.

[blopker]

Here you go: #358

The build times are longer, but how often are builds run? It looks like around monthly. As an added benefit, with the optimization flag we found an issue with the current images. If there's another issue at some point, running the tests may catch it.

[blopker]

I looked around a little and it turns out Travis CI has granted some open source projects more concurrent builders. WordPress has 15: https://make.wordpress.org/core/2017/05/18/increased-concurrent-jobs-on-travis-ci-builds/

Here travis-ci/travis-ci#4668 they mention sending an email to [email protected] to ask about raising the limit. Might be worth a try.

[tianon]

Travis is only used here for basic smoke tests to ensure the real builds will work properly (and is not the primary issue with build times here, although it's certainly a strong consideration). The real builds will take closer to that 12 hour mark (if not longer, since that official infra often has io contention issues).

[blopker]

Oh, that's too bad. I didn't know there was another build system. I usually just use Travis to push images once built: https://docs.travis-ci.com/user/docker/

May be worth asking support for more workers anyway.

I do want to note that I just rolled out a Python 2.7 with optimizations to a production system at work (not Docker though) and we saw a visible minimum 10% performance gain across all endpoints. Endpoints with heavy database access saw the least gain, of course.

If this change doesn't go through I'll definitely set up optimized images for internal use, but I'd love to share it with the world :)

[gaby]

Any updates regarding this PR? It would be great addition to have optimized Images!

[yosifkit]

Closing since we cannot handle the ~16x on build times.

Since these optimizations are only based on running the test suite it would make sense that they could be all pre-created and released with the source, rather than having distributions recreate what is, in essence, a static optimization.

[gpshead]

See the comment on the issue, you don't need to use the default overly-long PROFILE_TASK setting to see a large benefit.