Skip to content

Commit

Permalink
Added sample iOS Project with test plans
Browse files Browse the repository at this point in the history 
* added feature doc template
* refactored Xctestru
* added ios_test_plan.yml
  • Loading branch information
@zuziaka
zuziaka committed Oct 30, 2020
1 parent c45adcb commit a9f6144
Show file tree
Hide file tree
Showing 7 changed files with 293 additions and 121 deletions.
187 changes: 187 additions & 0 deletions docs/feature/ios_test_plans.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,187 @@
# Flow

Flow starts by parsing .xctestrun file.
Search for: `__xctestrun_metadata__` key.

```xml
<key>__xctestrun_metadata__</key>
<dict>
<key>FormatVersion</key>
<integer>1</integer>
</dict>
```

- **FormatVersion: `1` -** old version of .xctestrun
- **FormatVersion: `2` -** the newest version with test plans
- If format is different than 1 or 2 throw an error.

---

### FormatVersion: 1

Any other key than metadata should have corresponding **TestTarget** dictionary. In example below `EarlGreyExampleSwiftTests` has a **TestTarget** dictionary.

```xml
<plist version="1.0">
<dict>
<key>EarlGreyExampleSwiftTests</key>
<dict>
<key>BlueprintName</key>
<string>EarlGreyExampleSwiftTests</string>
...
</dict>
<key>__xctestrun_metadata__</key>
<dict>
<key>FormatVersion</key>
<integer>1</integer>
</dict>
</dict>
</plist>
```

### FormatVersion: 2

In this version, XML contains two keys: `TestConfigurations` and `TestPlan` in addition to `__xctestrun_metadata__`.

`TestPlan` is just dictionary containing basic informations about current **TestPlan.** We can ignore it.

`TestConfigurations` is an array of different test configurations. Test configuration contains name property and array of TestTargets.

```xml
<plist version="1.0">
<dict>
<key>Name</key>
<string>pl</string> <!-- Name property -->
<key>TestTargets</key>
<array>
<dict>
<key>BlueprintName</key>
<string>UITests</string>
<!-- Test target and Test configuration properties go here -->
</dict>
<dict>
<key>BlueprintName</key>
<string>SecondUITests</string>
<!-- Test target and Test configuration properties go here -->
</dict>
</array>
</dict>
</plist>
```

Each configuration may contain different Environment Variables, languages, regions or any other properties. Those properties are stored under TestTarget.

Currently **FTL** doesn't support specifying TestConfiguration for test execution.

If there is more than one configuration FTL will probably choose one arbitrarily.

For now Flank will allow specifying which test configuration should run with `only-test-configuration` argument.

---

# Running test plan locally

## Build Xcode project

To build example project run command below.

```bash
xcodebuild build-for-testing \
-allowProvisioningUpdates \
-project "FlankMultiTestTargetsExample.xcodeproj" \
-scheme "AllTests" \ #Scheme should have test plans enabled
-derivedDataPath "build_testplan_device" \
-sdk iphoneos | xcpretty
```

This command will generate directory: **Debug-iphoneos** containing binaries and .xctestrun file for each TestPlan.

In this example scheme `AllTests` has have only one test plan: **AllTests** with two test configurations: `pl` and `en`.

**Test Plan** contains two **Test Targets: `UITests` and `SecondUITests`**
Outputted .xctestrun should looks like this:

```xml
<plist version="1.0">
<dict>
<key>TestConfigurations</key>
<array>
<dict>
<key>Name</key>
<string>en</string>
<key>TestTargets</key> <!-- Test Targets for `en` test configuration -->
<array>
<dict>
<key>BlueprintName</key>
<string>UITests</string>
<key>TestLanguage</key>
<string>en</string>
<key>TestRegion</key>
<string>GB</string> <!-- Language and region -->
<!-- ... Other properties -->
</dict>
<dict>
<key>BlueprintName</key>
<string>SecondUITests</string>
<key>TestLanguage</key>
<string>en</string>
<key>TestRegion</key>
<string>GB</string> <!-- Language and region -->
<!-- ... Other properties -->
</dict>
</array>
</dict>
<dict>
<key>Name</key>
<string>pl</string>
<key>TestTargets</key> <!-- Test Targets for `pl` test configuration -->
<array>
<dict>
<key>BlueprintName</key>
<string>UITests</string>
<key>TestLanguage</key>
<string>pl</string>
<key>TestRegion</key>
<string>PL</string> <!-- Language and region -->
<!-- ... Other properties -->
</dict>
<dict>
<key>BlueprintName</key>
<string>SecondUITests</string>
<key>TestLanguage</key>
<string>pl</string>
<key>TestRegion</key>
<string>PL</string> <!-- Language and region -->
<!-- ... Other properties -->
</dict>
</array>
</dict>
</array>
<key>TestPlan</key>
<dict>
<key>IsDefault</key>
<true/>
<key>Name</key>
<string>AllTests</string>
</dict>
<key>__xctestrun_metadata__</key>
<dict>
<key>FormatVersion</key>
<integer>2</integer>
</dict>
</dict>
</plist>
```

## Running tests on a local device

After generating binaries and .xctestrun file we can run tests using command.

```bash
xcodebuild test-without-building \
-xctestrun "build_testplan_device/Build/Products/testrun.xctestrun" \
-destination "platform=iOS,id=00008030-000209DC1A50802E" \
-only-test-configuration pl | xcpretty
```

Option: `-only-test-configuration pl` allows to specify which test configuration should Xcode run.
2 changes: 1 addition & 1 deletion test_runner/src/main/kotlin/ftl/gc/GcIosTestMatrix.kt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ import ftl.args.IosArgs
import ftl.gc.android.mapGcsPathsToFileReference
import ftl.gc.android.mapToIosDeviceFiles
import ftl.ios.Xctestrun
import ftl.ios.Xctestrun.toByteArray
import ftl.ios.xctest.toByteArray
import ftl.run.exception.FlankGeneralError
import ftl.util.ShardCounter
import ftl.util.join
Expand Down
127 changes: 7 additions & 120 deletions test_runner/src/main/kotlin/ftl/ios/Xctestrun.kt
View file
Original file line number Diff line number Diff line change
@@ -1,133 +1,20 @@
package ftl.ios

import com.dd.plist.NSArray
import com.dd.plist.NSDictionary
import com.dd.plist.NSString
import com.dd.plist.PropertyListParser
import com.google.common.annotations.VisibleForTesting
import ftl.run.exception.FlankGeneralError
import java.io.ByteArrayOutputStream
import ftl.ios.xctest.findTestNames
import ftl.ios.xctest.parseToNSDictionary
import ftl.ios.xctest.rewriteXcTestRun
import java.io.File
import java.nio.file.Paths

typealias XctestrunMethods = Map<String, List<String>>

object Xctestrun {

private fun String.isMetadata(): Boolean {
return this.contentEquals("__xctestrun_metadata__")
}
fun parse(xctestrun: String): NSDictionary = parseToNSDictionary(File(xctestrun))

// Parses all tests for a given target
private fun testsForTarget(testDictionary: NSDictionary, testTarget: String, testRoot: String): List<String> {
if (testTarget.isMetadata()) return emptyList()
val skipTestIdentifiers: NSArray? = testDictionary["SkipTestIdentifiers"] as NSArray?
val skipTests: List<String> = skipTestIdentifiers?.array?.mapNotNull { (it as NSString?)?.content } ?: listOf()
val productPaths = testDictionary["DependentProductPaths"] as NSArray
for (product in productPaths.array) {
val productString = product.toString()
if (productString.contains("/$testTarget.xctest")) {
val binaryRoot = productString.replace("__TESTROOT__/", testRoot)
println("Found xctest: $binaryRoot")
fun parse(xctestrun: ByteArray): NSDictionary = parseToNSDictionary(xctestrun)

val binaryName = File(binaryRoot).nameWithoutExtension
val binaryPath = Paths.get(binaryRoot, binaryName).toString()
fun findTestNames(xctestrun: String): List<String> = findTestNames(File(xctestrun))

val tests = (Parse.parseObjcTests(binaryPath) + Parse.parseSwiftTests(binaryPath)).distinct()

return tests.minus(skipTests)
}
}

throw FlankGeneralError("No tests found")
}

// https://github.com/google/xctestrunner/blob/51dbb6b7eb35f2ed55439459ca49e06992bc4da0/xctestrunner/test_runner/xctestrun.py#L129
private const val onlyTestIdentifiers = "OnlyTestIdentifiers"

// Rewrites tests so that only the listed tests execute
private fun setOnlyTestIdentifiers(testDictionary: NSDictionary, tests: Collection<String>) {
val nsArray = NSArray(tests.size)
tests.forEachIndexed { index, test -> nsArray.setValue(index, test) }

while (testDictionary.containsKey(onlyTestIdentifiers)) {
testDictionary.remove(onlyTestIdentifiers)
}

testDictionary[onlyTestIdentifiers] = nsArray
}

fun parse(xctestrun: String): NSDictionary {
return parse(File(xctestrun))
}

// Parses xctestrun file into a dictonary
fun parse(xctestrun: File): NSDictionary {
val testrun = xctestrun.canonicalFile
if (!testrun.exists()) throw FlankGeneralError("$testrun doesn't exist")

return PropertyListParser.parse(testrun) as NSDictionary
}

fun parse(xctestrun: ByteArray): NSDictionary {
return PropertyListParser.parse(xctestrun) as NSDictionary
}

fun findTestNames(xctestrun: String): XctestrunMethods {
return findTestNames(File(xctestrun))
}

fun findTestNames(testTarget: String, xctestrun: String): List<String> =
findTestNamesForTarget(
testTarget = testTarget,
xctestrun = File(xctestrun)
)

private fun findTestNames(xctestrun: File): XctestrunMethods =
parse(xctestrun).allKeys().associate { testTarget ->
testTarget to findTestNamesForTarget(
testTarget = testTarget,
xctestrun = xctestrun
)
}

private fun findTestNamesForTarget(
testTarget: String,
xctestrun: File
): List<String> =
testsForTarget(
testDictionary = parse(xctestrun)[testTarget]
as? NSDictionary
?: throw FlankGeneralError("XCTestrun does not contain $testTarget test target."),
testRoot = xctestrun.parent + "/",
testTarget = testTarget
).distinct()

fun rewrite(xctestrun: String, methods: List<String>): ByteArray {
val xctestrunFile = File(xctestrun)
val methodsToRun = findTestNames(xctestrunFile).mapValues { (_, list) -> list.filter(methods::contains) }
return rewrite(parse(xctestrunFile), methodsToRun)
}

@VisibleForTesting
internal fun rewrite(root: NSDictionary, data: XctestrunMethods): ByteArray {
val rootClone = root.clone()

for (testTarget in rootClone.allKeys()) {
if (testTarget.isMetadata()) continue
val methods = data[testTarget]
if (methods != null) {
val testDictionary = rootClone[testTarget] as NSDictionary
setOnlyTestIdentifiers(testDictionary, methods)
}
}

return rootClone.toByteArray()
}

fun NSDictionary.toByteArray(): ByteArray {
val out = ByteArrayOutputStream()
PropertyListParser.saveAsXML(this, out)
return out.toByteArray()
}
fun rewrite(root: NSDictionary, methods: Collection<String>) = rewriteXcTestRun(root, methods)
}
51 changes: 51 additions & 0 deletions test_runner/src/main/kotlin/ftl/ios/xctest/FindTestNames.kt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
package ftl.ios.xctest

import com.dd.plist.NSArray
import com.dd.plist.NSDictionary
import com.dd.plist.NSObject
import com.dd.plist.NSString
import ftl.ios.Parse
import ftl.run.exception.FlankGeneralError
import java.io.File
import java.nio.file.Paths

// Finds tests in a xctestrun file
internal fun findTestNames(xctestrun: File): List<String> =
parseToNSDictionary(xctestrun).run {
val testRoot = xctestrun.parent + "/"
allKeys().map { testTarget ->
(get(testTarget) as NSDictionary).findTestsForTarget(
testRoot = testRoot,
testTarget = testTarget
)
}.flatten().distinct()
}

private fun NSDictionary.findTestsForTarget(testTarget: String, testRoot: String): List<String> =
if (testTarget.isMetadata()) emptyList()
else findXcTestTargets(testTarget)
?.run { findBinaryTests(testRoot) - findTestsToSkip() }
?: throw FlankGeneralError("No tests found")

private fun NSDictionary.findXcTestTargets(testTarget: String): NSObject? =
get("DependentProductPaths")
?.let { it as? NSArray }?.array
?.first { product -> product.toString().containsTestTarget(testTarget) }

private fun String.containsTestTarget(name: String): Boolean = contains("/$name.xctest")

private fun NSObject.findBinaryTests(testRoot: String): List<String> {
val binaryRoot = toString().replace("__TESTROOT__/", testRoot)
println("Found xctest: $binaryRoot")

val binaryName = File(binaryRoot).nameWithoutExtension
val binaryPath = Paths.get(binaryRoot, binaryName).toString()

return (Parse.parseObjcTests(binaryPath) + Parse.parseSwiftTests(binaryPath)).distinct()
}

private fun NSDictionary.findTestsToSkip(): List<String> =
get("SkipTestIdentifiers")
?.let { it as? NSArray }?.array
?.mapNotNull { (it as? NSString)?.content }
?: emptyList()
Loading

0 comments on commit a9f6144

Please sign in to comment.