swift test does not discover all tests

[glbrntt]

Description

On Linux platforms the test discovery mechanism does not discover all tests. Specifically, it does not discover tests which are declared in a parent class.

As an example:

import XCTest

class BaseTests: XCTestCase {
  var shouldPass: Bool { true }

  func testItShouldPass() throws {
    XCTAssertTrue(self.shouldPass)
  }
}

class InerhitedTests: BaseTests {
  override var shouldPass: Bool { false }
}

Should include two tests: BaseTests.testItShouldPass and InheritestTests.testItShouldPass.

On Linux only BestTests.testItShouldPass is discovered:

$ swift test --list-tests
Building for debugging...
Build complete! (0.27s)
MissingTestsTests.BaseTests/testItShouldPass

On macOS all tests are discovered:

$ swift test --list-tests
Building for debugging...
Build complete! (0.64s)
MissingTestsTests.BaseTests/testItShouldPass
MissingTestsTests.InerhitedTests/testItShouldPass

As expected, the same is true when running the tests: 2 tests are run on macOS and only 1 on Linux.

Expected behavior

I would expect the output of swift test --list-tests to be the same on macOS and Linux.

Actual behavior

Tests which run on macOS are silently not run on Linux when using test discovery.

Steps to reproduce

See example in description.

Swift Package Manager version/commit hash

No response

Swift & OS version (output of swift --version && uname -a)

Swift version 5.8-dev (LLVM 278d67f38c6a910, Swift ee312bc1e20eb01)
Target: aarch64-unknown-linux-gnu
Linux 2b30d1439f50 5.10.104-linuxkit #1 SMP PREEMPT Thu Mar 17 17:05:54 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

This also reproduces with older Linux toolchains.

[tomerd]

radar://59655518

[art-divin]

👋🏻 Hello,

this issue is still happening, I have faced it during implementation of Sourcery on Linux, where tests are discovered on macOS, but not on Linux.

Any pointers anyone? I'd appreciate it a bunch 🙏🏻

[neonichu]

Looks like this was supposed to be solved by #5637 -- which version of SwiftPM were you using?

[art-divin]

👋🏻 Hello @neonichu , thank you very much for your response!

which version of SwiftPM were you using?

I have tried to use the latest development snapshot from 12th of August, 2023, facing the same issue there.

I have also asked this question on Swift forums and received a useful reply, related to specifics of Sourcery library, which relies on Quick framework, which has this issue, where tests are not discovered, even though test (<#something#>Spec) classes are subclassed from XCTestCase on non-Darwin platforms [1].

I am trying to make it work according to what's documented there, but I wanted to run SPM myself from sources to debug and probably fix this issue, and could not, hitting "PackageDescription" import failing [2] when calling a compiled package-manager with debugger attached - is there any place I could read up on how to run SPM with debugger attached, like, a configuration of some sort?

Notes

[1] - public typealias QuickSpecBase = XCTestCase
[2] -

error: 'sourcery': Invalid manifest (compiled with: ["/Applications/Xcode-14.3.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swiftc", "-vfsoverlay", "/var/folders/90/8tcwdk19055d0rrgfp00h25h0000gn/T/TemporaryDirectory.EfZdEd/vfs.yaml", "-L", "/Users/art-divin/Documents/Projects/swift-package-manager/.build/arm64-apple-macosx/debug", "-lPackageDescription", "-Xlinker", "-rpath", "-Xlinker", "/Users/art-divin/Documents/Projects/swift-package-manager/.build/arm64-apple-macosx/debug", "-target", "arm64-apple-macosx13.0", "-sdk", "/Applications/Xcode-14.3.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk", "-F", "/Applications/Xcode-14.3.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/Library/Frameworks", "-I", "/Applications/Xcode-14.3.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/usr/lib", "-L", "/Applications/Xcode-14.3.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/usr/lib", "-swift-version", "5", "-I", "/Users/art-divin/Documents/Projects/swift-package-manager/.build/arm64-apple-macosx/debug", "-sdk", "/Applications/Xcode-14.3.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk", "-package-description-version", "5.6.0", "/Users/art-divin/Documents/Projects/Sourcery/Package.swift", "-Xfrontend", "-disable-implicit-concurrency-module-import", "-Xfrontend", "-disable-implicit-string-processing-module-import", "-o", "/var/folders/90/8tcwdk19055d0rrgfp00h25h0000gn/T/TemporaryDirectory.2jdc44/sourcery-manifest"])
/Users/art-divin/Documents/Projects/swift-package-manager/.build/arm64-apple-macosx/debug/PackageDescription.build/module.modulemap:2:12: error: header '/Users/art-divin/Documents/Projects/swift-package-manager/.build/arm64-apple-macosx/debug/PackageDescription.build/PackageDescription-Swift.h' not found
    header "/Users/art-divin/Documents/Projects/swift-package-manager/.build/arm64-apple-macosx/debug/PackageDescription.build/PackageDescription-Swift.h"
           ^
/Users/art-divin/Documents/Projects/Sourcery/Package.swift:4:8: error: could not build Objective-C module 'PackageDescription'
import PackageDescription
       ^
error: fatalError

[neonichu]

cc @tomerd, it seems like #5637 may not have fixed this

[tomerd]

the fix (and logic) itself was done in https://github.com/apple/swift-tools-support-core/pull/335/files#diff-de74314b412b8d423fec461826c4ad64056d9d77e95c65ace4e27eb969beb8acR84

cc @stmontgomery since this is based on XCTest internals afaik

[stmontgomery]

My best guess is that this might still be happening only when using an "intermediary" testing library like Quick. Is the current logic in SwiftPM for using indexer data to find tests only examining the object files for the test target(s) in a particular package, or does this logic also scan the object files from package dependencies? If it's not scanning the object files for Quick package too, it might not realize that the QuickSpec parent class itself inherits from XCTestCase and so any descendant of QuickSpec should also be considered a test class.

@tomerd I'm not deeply familiar with this logic in SwiftPM, but are you able to check on this?

[art-divin]

👋🏻 Hello @stmontgomery ,

thank you very much for your response!

I would like to add a couple of notes here:

1. I have tried adding a simple class MyTestCase: XCTestCase { func testFailingTest() { XCTFail("fail") } } to the existing file under Sourcery/Tests/Parsing/ComposerSpec.swift and swift test did discover this test. CI confirming my words can be found here.
2. While tinkering with bits and pieces of information I have gathered, I have done the following:
   2.1. ran swift test --generate-linuxmain under macOS (where test are discovered)
   2.2 committed changes and pulled from ubuntu
   2.3 removed autogenerated code from LinuxMain.swift and replaced it with contents proposed here as an example, using actual test class names.
   2.4 ran swift test -> it lead to tests being executed! ✅

I would love to just debug this myself since I have already cloned SPM and tried to run it against Sourcery' Package.swift, but as mentioned above, I see a runtime compilation error with that import PackageDescription. Any pointers would be appreciated, I'd love to fix it myself in SPM ❤️

[MaxDesiatov]

If you're building SwiftPM in Xcode and seeing import PackageDescription errors, select the whole package in the list of schemes and build that first, which will build the required PackageDescription library. Then you can switch to schemes with binary executables, such as swift-test, and run those.