Integration test test_llvm_x86_01 fails on MacOS

[brianhuffman]

As of eeef9a1, running intTests gives me the following result:

tests passed: 70 /       71
failed: test_llvm_x86_01

Aren't we running these tests on CI yet?

[atomb]

Hmm. The most recent run on Travis succeeded: https://travis-ci.org/github/GaloisInc/saw-script/jobs/689333557#L543

[brianhuffman]

Ok, I see why it's failing. This is apparently the only regression test that uses yasm; the test had failed because it couldn't find the yasm binary on my system.

It might be nice to add some text to the README file listing what external programs are necessary to run the tests.

[brianhuffman]

I installed yasm, but the test is still failing. Here's the contents of results.xml after running ./runtests.sh test_llvm_x86_01:

<?xml version='1.0'?>
<testsuites errors="1" tests="       1" time="0">
 <testsuite name="SAWScript Integration Tests">
  <testcase name="test_llvm_x86_01" time="0"><failure><![CDATA[
set -e
+ set -e

yasm -felf64 test.S
+ yasm -felf64 test.S
ld test.o -o test
+ ld test.o -o test
ld: warning: No version-min specified on command line
ld: warning: -arch not specified
ld: warning: -macosx_version_min not specified, assuming 10.11
ld: warning: ignoring file test.o, file was built for unsupported file format ( 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 ) which is not the architecture being linked (x86_64): test.o
Undefined symbols for architecture x86_64:
  "_main", referenced from:
     implicit entry/start for main executable
ld: symbol(s) not found for inferred architecture x86_64
]]></failure></testcase>
 </testsuite>
</testsuites>

So is it futile to even try running this test on MacOS? If so, are we ok with having some tests that fail to run on certain platforms? Should we check in a linux-built copy of the executable binary that we're verifying in this test?

[atomb]

I'd advocate for checking in the object file. Otherwise, even on Linux, the test could be quite fragile. For the test suite, I'm also in favor of checking in bitcode files, as long as we also have an easy way to regenerate them.

[brianhuffman]

This test seems to be working now, probably due to #984.