i#4347: Fix macOS build OOM on Travis

[abhinav92003]

macOS build fail on Travis: https://travis-ci.com/github/DynamoRIO/dynamorio/jobs/356507267

Fixes: #4347

[abhinav92003]

I noticed that the xcode version in the logs is 9.4.1 instead of the updated 12 or even the previous 11.3. xcode9.4 is the default version as per https://docs.travis-ci.com/user/reference/osx/#macos-version.

InstalledDir: /Applications/Xcode-9.4.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
...

-- Check for working C compiler: /Applications/Xcode-9.4.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang
-- Check for working C compiler: /Applications/Xcode-9.4.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -- works

[derekbruening]

I put in the 11.3 upgrade in February and it certainly took effect back then as you can see in PR #4096 https://travis-ci.com/github/DynamoRIO/dynamorio/jobs/287132339

InstalledDir: /Applications/Xcode-11.3.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

There were failures with the default, so you would think we'd notice if it regressed at some point back to 9.4.

[derekbruening]

I checked a few builds and it looks like Travis made some change which caused the regression between our #4149 and #4150 on Mar 1-2:

• i#4134: DrBBDup: basic block duplication #4140 11.3 Mar 1
• i#3917: Fix gcc 9.2.1 build errors #4149 11.3 Mar 1
• i#4083 func trace: Add func_view tool #4150 9.4 Mar 2

[abhinav92003]

We're really using xcode11.3.1 now.

InstalledDir: /Applications/Xcode-11.3.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

But the initial OOM error is also back: https://travis-ci.com/github/DynamoRIO/dynamorio/jobs/356568480

====> FAILURE in debug-internal-64 <====
debug-internal-64: **** 7 build errors ****
	critical: /.../dynamorio/core/arch/x86/x86.asm:2517: out of memory!
	make[2]: *** [core/CMakeFiles/unit_tests.dir/arch/x86/x86.asm.o] Error 2
	make[1]: *** [core/CMakeFiles/unit_tests.dir/all] Error 2
	critical: /.../dynamorio/core/arch/x86/x86.asm:2517: out of memory!
	make[2]: *** [core/CMakeFiles/dynamorio.dir/arch/x86/x86.asm.o] Error 2
	make[1]: *** [core/CMakeFiles/dynamorio.dir/all] Error 2
	make: *** [all] Error 2
====> FAILURE in release-external-64 <====
release-external-64: **** 7 build errors ****
	critical: /.../dynamorio/core/arch/x86/x86.asm:2517: out of memory!
	make[2]: *** [core/CMakeFiles/unit_tests.dir/arch/x86/x86.asm.o] Error 2
	make[1]: *** [core/CMakeFiles/unit_tests.dir/all] Error 2
	critical: /.../dynamorio/core/arch/x86/x86.asm:2517: out of memory!
	make[2]: *** [core/CMakeFiles/dynamorio.dir/arch/x86/x86.asm.o] Error 2
	make[1]: *** [core/CMakeFiles/dynamorio.dir/all] Error 2
	make: *** [all] Error 2

[abhinav92003]

Update on the attempts to fix this OOM so far.

We expected that we were using xcode11.3 for all builds, but we were actually using the default xcode9.4 due to some missing Travis configs which I added in f3b896b. This didn't fix the OOM though.

I tried using the latest xcode12 which is in beta right now (see 7467174), and also xcode11.5 which is the latest Golden release (see dcebe86). In both cases, we're seeing new errors as @derekbruening observed in #4344.

I also tried adding some missing options in Travis config that were there in last successful build with xcode11.3 (see 5a2056e), but that didn't help either.

dmesg is not producing any output to help debug the OOM (see 674997b)

@derekbruening Do you think it's worth fixing #4344 and hoping the new xcode resolves the OOM too?

[abhinav92003]

Tried xcode8.3 as it was documented to help with OOM issues in some newer versions (see b2221c5). Not sure if we want to use such an old xcode, but it helps to some extent. The build does proceed further than before, but fails due to a different reason: https://travis-ci.com/github/DynamoRIO/dynamorio/jobs/357032530
/.../dynamorio/core/arch/x86_code_test.c:171:5: error: invalid output constraint '=v' in asm

[derekbruening]

We expected that we were using xcode11.3 for all builds, but we were actually using the default xcode9.4 due to some missing Travis configs which I added in f3b896b. This didn't fix the OOM though.

Is that config documented by Travis? Any idea why we did not need it initially when xcode11.3 was in fact being used?

@derekbruening Do you think it's worth fixing #4344 and hoping the new xcode resolves the OOM too?

Yes, our own machines are going to have recent versions so we'll have to solve it anyway. If you would like to take over #4344 please go ahead. I will put notes there on what little I have tried so far.

[abhinav92003]

Travis macOS builds have started passing again, of their own accord. All of the following were failing previously.

Green on latest submitted PR in master: https://travis-ci.com/github/DynamoRIO/dynamorio/jobs/356922047
Green on one of my pending PR #4341: https://travis-ci.com/github/DynamoRIO/dynamorio/jobs/357179016
Green on this PR: https://travis-ci.com/github/DynamoRIO/dynamorio/jobs/357182309

I think 7467174 is still relevant so that we use a more recent version of xcode. I'll send it in a separate PR.

Is that config documented by Travis? Any idea why we did not need it initially when xcode11.3 was in fact being used?

It's the same osx_image option but in the jobs section. https://docs.travis-ci.com/user/multi-os/#example-multi-os-build-matrix shows a build matrix example with osx_image.

[abhinav92003]

Also see #4354 for an experiment that shows macOS builds passing on reverting last 3 commits and re-committing them one at a time. These commits were failing on master when they were submitted.

[derekbruening]

Is that config documented by Travis? Any idea why we did not need it initially when xcode11.3 was in fact being used?

It's the same osx_image option but in the jobs section. https://docs.travis-ci.com/user/multi-os/#example-multi-os-build-matrix shows a build matrix example with osx_image.

OK, I saw a commit go by with sthg like group: stable and that was what I thought was undocumented.