Support for Xcode 11 #32

thibault-ml · 2019-07-25T18:34:13Z

Xcode 11 uses a new format for xctest (version 3), which doesn't have a TestSummary.plist file, and uses Zstandard-compressed "database-like" files.

Xcode 11 Release Notes state:

The format of result bundles changed in Xcode 11. A result bundle is an asset produced by Xcode 11 with the xcresult file extension that contains information about the build, tests, code coverage, and more. Any xcresult files produced with Xcode 10 or earlier cannot be read by Xcode 11. A result bundle can be produced by passing -resultBundlePath ./Example.xcresult to an xcodebuild invocation and the Example.xcresult can then be opened in Xcode. Xcode also creates result bundles in Derived Data. The current result bundle version number is 3, which can be specified by passing the xcodebuild flag -resultBundleVersion 3. Version 3 is the default in Xcode 11, but it is still recommended for automation to explicitly pass the flag, so that any potential future versions that become the default do not cause issues to existing tools. Result bundles can be inspected using xcresulttool. A JSON representation of the root object of the result bundle can be exported using xcrun xcresulttool get --format json --path ./Example.xcresult and any nested object, identified by its reference found in the JSON output, can be exported by adding the flag --id REF. xcresulttool also provides the description of its format using xcrun xcresulttool formatDescription. (41633595)

Basically, to obtain results from an Xcode 11 xcresult, one would have to do the following:

// The output of this command will be useful to automatically parse the output of the next commands
$ xcrun xcresulttool formatDescription get --format json
[...]

$ xcrun xcresulttool get --format json --path Project.xcresult
[...]
          "testsRef" : {
            "_type" : {
              "_name" : "Reference"
            },
            "id" : {
              "_type" : {
                "_name" : "String"
              },
              "_value" : "0~YVEC1htpkY3jLmeu4DB8ah-sJueP7_vUdqA2rL1_R3xT_aftqqEgGUSCHZ0mmpojBSyy4mEL4LX2xyFlfr4pEQ=="
            },
[...]

// We take the value of the `testsRef` key (if any) for each `ActionRecord` entry
$ xcrun xcresulttool get --format json --path Project.xcresult --id '0~YVEC1htpkY3jLmeu4DB8ah-sJueP7_vUdqA2rL1_R3xT_aftqqEgGUSCHZ0mmpojBSyy4mEL4LX2xyFlfr4pEQ=='
{
  "_type" : {
    "_name" : "ActionTestPlanRunSummaries"
  },
  "summaries" : {
    "_type" : {
      "_name" : "Array"
    },
    "_values" : [
[...]

The text was updated successfully, but these errors were encountered:

snatchev · 2019-08-14T13:35:53Z

hi @thibault-ml thanks for bringing this up. I will take a look at this once I get some of these JSON files.

thibault-ml · 2019-08-14T13:40:24Z

Thanks!

Google already parses them:
https://chromium.googlesource.com/chromium/src/+/refs/heads/master/ios/build/bots/scripts/
https://github.com/chromium/chromium/tree/master/ios/build/bots/scripts

Bitrise too: https://github.com/bitrise-steplib/steps-deploy-to-bitrise-io/tree/79d467e68e077eddf8f67277a212e42d0b4cd52a/test/converters/xcresult3

And XCResultKit too: https://github.com/davidahouse/XCResultKit

joshdholtz · 2019-09-04T03:06:07Z

@thibault-ml Looking into! Will hopefully have something for you soon 😊

joshdholtz · 2019-09-04T20:09:34Z

@thibault-ml Hey! Do you happen to have a xcresult file you could send me? 😊 It would be good to have a legit file that I can test on 🙏

thibault-ml · 2019-09-06T15:45:26Z

@joshdholtz I'll have a look if we can, not sure 100% I'm at liberty to just yet. Will have a look :-)

joshdholtz · 2019-09-06T16:06:33Z

@thibault-ml If you don't want to send but are able to run my PR that would also be 💯 😊 The PR works for me on my simple test use cases (I don't have any huge test suites). I got a few fixes to do on it and then will merge.

juliangrosshauser · 2019-09-06T16:19:39Z

@joshdholtz Here's a .xcresult directory of one of our PSPDFKit test suites. Should contain over 1000 test results:

PSPDFKit.xcresult.zip

Your PR fails with:

bundler: failed to load command: trainer (/Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bin/trainer)
NoMethodError: [!] undefined method `[]' for nil:NilClass
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:308:in `initialize'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:337:in `new'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:337:in `block in initialize'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:336:in `map'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:336:in `initialize'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:238:in `new'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:238:in `initialize'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:209:in `new'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:209:in `initialize'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:181:in `new'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:181:in `block in initialize'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:180:in `map'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:180:in `initialize'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/test_parser.rb:141:in `new'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/test_parser.rb:141:in `parse_xcresult'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/test_parser.rb:61:in `initialize'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/test_parser.rb:47:in `new'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/test_parser.rb:47:in `block in auto_convert'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/test_parser.rb:33:in `each'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/test_parser.rb:33:in `auto_convert'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/commands_generator.rb:34:in `block (2 levels) in run'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:178:in `call'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:153:in `run'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/gems/commander-fastlane-4.4.6/lib/commander/runner.rb:476:in `run_active_command'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/gems/fastlane-2.130.0/fastlane_core/lib/fastlane_core/ui/fastlane_runner.rb:76:in `run!'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/gems/commander-fastlane-4.4.6/lib/commander/delegates.rb:15:in `run!'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/commands_generator.rb:40:in `run'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/commands_generator.rb:10:in `start'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/bin/trainer:6:in `<top (required)>'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bin/trainer:23:in `load'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bin/trainer:23:in `<top (required)>'

joshdholtz · 2019-09-06T16:53:08Z

@juliangrosshauser Thanks for sending that over! I had a feeling my PR would need some tweaking for something a little more complex 🙃 I will make sure it works with you sent over ❤️

joshdholtz · 2019-09-07T01:35:28Z

@juliangrosshauser Got it solved! Thanks sending this 🙌

joshdholtz · 2019-09-07T03:09:24Z

@juliangrosshauser Alright, PR should be good to go again if you'd like to test when you get some time. I was able to get a successful output from the xcresults that you sent me 👌 I made sure all the data is getting pulled out safely now so that shouldn't happen anymore 😇

juliangrosshauser · 2019-09-07T20:18:25Z

@joshdholtz Great work! Thank you so much, really appreciate it!

I tested your PR and it works flawlessly now. The only problematic .xcresult directory I could find it this one: PSPDFKit-1-test-failure.zip

It contains 1008 test results. 1007 tests succeed, but the testStyledStringToPDFAsFileURL test fails. The problem is that the resulting XML file contains 1008 successful tests, but no failures.

joshdholtz · 2019-09-09T12:06:35Z

Oh boy 😱 I found the reason. The testCaseName name in the xcresulttool JSON output is something not as straight forward to match on 🤔 Will work on a solution though

"producingTarget" : {
    "_type" : {
        "_name" : "String"
    },
    "_value" : "PSPDFKitTests"
},
"testCaseName" : {
    "_type" : {
        "_name" : "String"
    },
    "_value" : "-[PSPDFProcessorTests testStyledStringToPDFAsFileURL]"
}

joshdholtz · 2019-09-09T12:35:32Z

@juliangrosshauser Thanks for testing again! Here is a commit for fixing that 😊 3f88286

juliangrosshauser · 2019-09-09T20:14:54Z

@joshdholtz I tested your newest changes with Objective-C and Swift failures. Looks like everything works now! 🎉 Thank you again!

joshdholtz · 2019-09-09T20:23:06Z

@juliangrosshauser Thanks so much for feeding me files needed to make this change 😊 I will merge the PR and get a new release out today at some point 💪🚀

vrutberg · 2019-09-18T14:01:03Z

Thank you guys for working on this!

Is the correct way to use this to specify the -resultBundlePath to xcodebuild, like this, when invoking scan?

xcargs: "-resultBundlePath ./TestSummaries.xcresult"

joshdholtz · 2019-09-20T01:50:16Z

@vrutberg I think that should work! But you should also be able to do...

scan(
  result_bundle: true,
  output_directory: "./" # optional 
)

diogot mentioned this issue Aug 13, 2019

Investigate Xcode 11 diogot/xcode-rakelib#2

Closed

snatchev self-assigned this Aug 14, 2019

hakota mentioned this issue Aug 20, 2019

'SCAN_GENERATED_PLIST_FILE ' lane context is nil with Xcode 11 Beta 5 fastlane/fastlane#15209

Closed

4 tasks

joshdholtz mentioned this issue Sep 4, 2019

Add support for Xcode 11 new xcresult format #34

Merged

7 tasks

joshdholtz closed this as completed in #34 Sep 10, 2019

babbage mentioned this issue Sep 20, 2019

Update for changed .xcresult format in Xcode 11 Antondomashnev/FBSnapshotsViewer#80

Open

lyndsey-ferguson mentioned this issue Oct 1, 2019

collating test_result bundles in Xcode 11 crashes lyndsey-ferguson/fastlane-plugin-test_center#145

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for Xcode 11 #32

Support for Xcode 11 #32

thibault-ml commented Jul 25, 2019

snatchev commented Aug 14, 2019

thibault-ml commented Aug 14, 2019

joshdholtz commented Sep 4, 2019

joshdholtz commented Sep 4, 2019

thibault-ml commented Sep 6, 2019

joshdholtz commented Sep 6, 2019

juliangrosshauser commented Sep 6, 2019

joshdholtz commented Sep 6, 2019

joshdholtz commented Sep 7, 2019

joshdholtz commented Sep 7, 2019

juliangrosshauser commented Sep 7, 2019

joshdholtz commented Sep 9, 2019

joshdholtz commented Sep 9, 2019

juliangrosshauser commented Sep 9, 2019

joshdholtz commented Sep 9, 2019

vrutberg commented Sep 18, 2019

joshdholtz commented Sep 20, 2019

Support for Xcode 11 #32

Support for Xcode 11 #32

Comments

thibault-ml commented Jul 25, 2019

snatchev commented Aug 14, 2019

thibault-ml commented Aug 14, 2019

joshdholtz commented Sep 4, 2019

joshdholtz commented Sep 4, 2019

thibault-ml commented Sep 6, 2019

joshdholtz commented Sep 6, 2019

juliangrosshauser commented Sep 6, 2019

joshdholtz commented Sep 6, 2019

joshdholtz commented Sep 7, 2019

joshdholtz commented Sep 7, 2019

juliangrosshauser commented Sep 7, 2019

joshdholtz commented Sep 9, 2019

joshdholtz commented Sep 9, 2019

juliangrosshauser commented Sep 9, 2019

joshdholtz commented Sep 9, 2019

vrutberg commented Sep 18, 2019

joshdholtz commented Sep 20, 2019