fix: #842 support for test targets flag in multiple testbundles xctest #1219

zuziaka · 2020-10-06T11:07:00Z

Fixes #842

Describe the bug

If the original .xctestrun has multiple XCTest entries, Tests included via --test-targets flag will be added to the OnlyTestIdentifiers array on every XCTest entry, not just the one(s) that actually includes that test.
__
Currently Flank rewrites whole .xctestrun file with methods specified --test-targets without checking whether those methods actually exists in XCTest. This solution works fine for a .xctestrun files containing only one XCTest.

Solution

Flank now rewrites .xctestrun with mapping correct methods names with specific XCTests.

Test Plan

How do we know the code works?

unit tested
new test artifacts with multiple XCTest entries

Checklist

Documented
Unit tested
Test artifacts added to TestProjects

zuziaka · 2020-10-06T11:08:52Z

test_runner/src/test/kotlin/ftl/ios/XctestrunTest.kt

@@ -17,6 +19,8 @@ import java.nio.file.Paths
 class XctestrunTest {

    private val swiftXctestrun = "$fixturesPath/EarlGreyExampleSwiftTests_iphoneos13.4-arm64e.xctestrun"
+    private val multipleTargetsSwiftXctestrun = "$fixturesPath/axel/AllTests_iphoneos13.7-arm64e.xctestrun"


@jan-gogo In order to test new functionality, I have to add some new fixtures, but we should wait for this PR
#1148, right?

@jan-gogo In order to test new functionality, I have to add some new fixtures, but we should wait for this PR
#1148, right?

Right

mergify · 2020-10-15T11:07:38Z

Title does not follow the guidelines of Conventional Commits.
Please adjust title before merge and use one of following prefix:

build - Changes that affect the build system or external dependencies (dependencies update)
ci - Changes to our CI configuration files and scripts (basically directory .github/workflows)
docs - Documentation only changes
feat - A new feature
fix - A bug fix
chore - Changes which does not touch the code (ex. manual update of release notes). It will not generate release notes changes
refactor - A code change that contains refactor
style - Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)
test - Adding missing tests or correcting existing tests and also changes for our test app
perf - A code change that improves performance (I do not think we will use it)

zuziaka · 2020-10-15T11:08:33Z

.gitignore

@@ -9,3 +9,4 @@ test_runner/flank/
 local.properties
 /report.json
 results
+xcuserdata/


User generated data for iOS projects (breakpoints, interface state etc.)

zuziaka · 2020-10-15T11:08:49Z

test_projects/ios/EarlGreyExample/EarlGreyExample.xcodeproj/project.pbxproj

@@ -509,9 +509,9 @@
 				COPY_PHASE_STRIP = NO;
 				DEBUG_INFORMATION_FORMAT = "dwarf-with-dsym";
 				DEFINES_MODULE = NO;
-				DEVELOPMENT_TEAM = AD2V26JBWL;
+				DEVELOPMENT_TEAM = L2UF9MLSM6;


Using GogoApps code signing

test_projects/ios/EarlGreyExample/ops.sh

codecov-io · 2020-10-15T11:13:45Z

Codecov Report

Merging #1219 into master will increase coverage by 0.07%.
The diff coverage is 95.12%.

@@             Coverage Diff              @@
##             master    #1219      +/-   ##
============================================
+ Coverage     79.65%   79.72%   +0.07%     
- Complexity      715      721       +6     
============================================
  Files           233      233              
  Lines          4399     4410      +11     
  Branches        756      757       +1     
============================================
+ Hits           3504     3516      +12     
  Misses          496      496              
+ Partials        399      398       -1

jan-goral · 2020-10-15T11:42:01Z

test_runner/build.gradle.kts

@@ -136,7 +136,7 @@ detekt {
    input = files("src/main/kotlin", "src/test/kotlin")
    config = files("../config/detekt.yml")
    parallel = true
-    failFast = true // fail build on any finding
+    failFast = false // fail build on any finding


What is the purpose of this change?

Sorry, changed this for testing - already removed

test_runner/src/main/kotlin/ftl/args/IosArgs.kt

test_runner/src/main/kotlin/ftl/ios/Xctestrun.kt

pawelpasterz · 2020-10-19T08:04:18Z

https://github.com/Flank/flank/pull/1219/checks?check_run_id=1273854203#step:5:147

test_runner/src/main/kotlin/ftl/args/IosArgs.kt

test_runner/src/main/kotlin/ftl/ios/Xctestrun.kt

piotradamczyk5

Just address @Sloox 'es comments and good to go

And also please check tests on Windows 😄
Probably you could skip them for now with TODO comment about solving them in #1134

…methods

- updated for using new test artefacts - resolved conflicts

- updated code signing in EarlGreyExample project - updated methods/test targets names

- added new iOS test project - created ops.sh scripts to generate test artifacts from it - adjusted fixture paths inside unit tests

- `findTestNamesForTarget` method Co-authored-by: Jan Góral <[email protected]>

Small XCTestRun's methods refactor Co-authored-by: Jan Góral <[email protected]>

Co-authored-by: pawelpasterz <[email protected]>

- refactored filterTests method

zuziaka commented Oct 6, 2020

View reviewed changes

jan-goral assigned zuziaka and jan-goral Oct 7, 2020

zuziaka force-pushed the #842-support-for-test-targets-flag-in-multiple-testbundles-xctest branch from 83503a6 to 960e9fc Compare October 14, 2020 08:35

zuziaka added Bug iOS labels Oct 15, 2020

zuziaka marked this pull request as ready for review October 15, 2020 11:06

bootstraponline force-pushed the #842-support-for-test-targets-flag-in-multiple-testbundles-xctest branch from d6e5864 to aafdb17 Compare October 15, 2020 11:06

zuziaka added Tiger 🐯 and removed Tiger 🐯 labels Oct 15, 2020

zuziaka requested a review from pawelpasterz October 15, 2020 11:07

zuziaka requested review from adamfilipow92 and piotradamczyk5 October 15, 2020 11:07

zuziaka commented Oct 15, 2020

View reviewed changes

zuziaka changed the title ~~#842 support for test targets flag in multiple testbundles xctest~~ fix: #842 support for test targets flag in multiple testbundles xctest Oct 15, 2020

jan-goral reviewed Oct 15, 2020

View reviewed changes

bootstraponline force-pushed the #842-support-for-test-targets-flag-in-multiple-testbundles-xctest branch 5 times, most recently from 9cf3f15 to a4f8412 Compare October 19, 2020 07:46

pawelpasterz reviewed Oct 19, 2020

View reviewed changes

test_runner/src/main/kotlin/ftl/args/IosArgs.kt Outdated Show resolved Hide resolved

test_runner/src/main/kotlin/ftl/ios/Xctestrun.kt Outdated Show resolved Hide resolved

test_runner/src/main/kotlin/ftl/ios/Xctestrun.kt Outdated Show resolved Hide resolved

bootstraponline force-pushed the #842-support-for-test-targets-flag-in-multiple-testbundles-xctest branch from 9de5636 to 3d2ab14 Compare October 19, 2020 10:27

jan-goral reviewed Oct 19, 2020

View reviewed changes

test_runner/src/main/kotlin/ftl/ios/Xctestrun.kt Outdated Show resolved Hide resolved

bootstraponline force-pushed the #842-support-for-test-targets-flag-in-multiple-testbundles-xctest branch from 191ea61 to 55cdaa0 Compare October 19, 2020 14:20

jan-goral approved these changes Oct 20, 2020

View reviewed changes

bootstraponline force-pushed the #842-support-for-test-targets-flag-in-multiple-testbundles-xctest branch from df76968 to 7410843 Compare October 20, 2020 09:56

jan-goral force-pushed the #842-support-for-test-targets-flag-in-multiple-testbundles-xctest branch from f1140d5 to 15105a8 Compare October 23, 2020 13:14

piotradamczyk5 approved these changes Oct 23, 2020

View reviewed changes

piotradamczyk5 self-requested a review October 23, 2020 13:36

piotradamczyk5 suggested changes Oct 23, 2020

View reviewed changes

bootstraponline force-pushed the #842-support-for-test-targets-flag-in-multiple-testbundles-xctest branch 3 times, most recently from dd4a268 to 97702ce Compare October 26, 2020 11:02

zuziaka and others added 19 commits October 26, 2020 11:57

XCTestRun.rewrite methods supports now multiple test-targets

83da53e

Usage of new Xctestrun.rewrite in IosRunCommand

8475584

Code formatting + Removed old implementations of IosArgs.filterTests …

b3d191b

…methods

Rebase from master

7b9cb10

- updated for using new test artefacts - resolved conflicts

Code cleaning

71f32bc

- updated code signing in EarlGreyExample project - updated methods/test targets names

Updated XCtestrun tests

4c8e96c

Reverted test changes in Debug.kt

7866caf

New test artifacts for iOS

aba2a7e

- added new iOS test project - created ops.sh scripts to generate test artifacts from it - adjusted fixture paths inside unit tests

Undo changes in build.gradle.kts

3e6ff93

Update test_runner/src/main/kotlin/ftl/ios/Xctestrun.kt

35878a5

- `findTestNamesForTarget` method Co-authored-by: Jan Góral <[email protected]>

Apply suggestions from code review

b406ba2

Small XCTestRun's methods refactor Co-authored-by: Jan Góral <[email protected]>

Fixed code formatting to pass :detekt

69c4a0e

Update test_runner/src/main/kotlin/ftl/ios/Xctestrun.kt

d8075f2

Co-authored-by: pawelpasterz <[email protected]>

Update test_runner/src/main/kotlin/ftl/ios/Xctestrun.kt

f669dea

Update test_runner/src/main/kotlin/ftl/args/IosArgs.kt

990864b

- refactored filterTests method

Fixed incorrect test artifacts path in ParseTest.kt

7a2d75b

Update test_runner/src/main/kotlin/ftl/ios/Xctestrun.kt

a52cc3d

Update test_runner/src/main/kotlin/ftl/ios/Xctestrun.kt

d211a21

Update test_runner/src/main/kotlin/ftl/ios/Xctestrun.kt

0f4021a

bootstraponline force-pushed the #842-support-for-test-targets-flag-in-multiple-testbundles-xctest branch from 97702ce to 0f4021a Compare October 26, 2020 11:57

jan-goral merged commit 8a81c2d into master Oct 26, 2020

jan-goral deleted the #842-support-for-test-targets-flag-in-multiple-testbundles-xctest branch October 26, 2020 13:08

pawelpasterz mentioned this pull request Dec 9, 2020

Output inconsistency (ios vs android) #1388

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: #842 support for test targets flag in multiple testbundles xctest #1219

fix: #842 support for test targets flag in multiple testbundles xctest #1219

zuziaka commented Oct 6, 2020 •

edited

Loading

zuziaka Oct 6, 2020 •

edited

Loading

jan-goral Oct 7, 2020

mergify bot commented Oct 15, 2020

zuziaka Oct 15, 2020

zuziaka Oct 15, 2020

codecov-io commented Oct 15, 2020 •

edited

Loading

jan-goral Oct 15, 2020

zuziaka Oct 16, 2020

pawelpasterz commented Oct 19, 2020

piotradamczyk5 left a comment

fix: #842 support for test targets flag in multiple testbundles xctest #1219

fix: #842 support for test targets flag in multiple testbundles xctest #1219

Conversation

zuziaka commented Oct 6, 2020 • edited Loading

Describe the bug

Solution

Test Plan

Checklist

zuziaka Oct 6, 2020 • edited Loading

Choose a reason for hiding this comment

jan-goral Oct 7, 2020

Choose a reason for hiding this comment

mergify bot commented Oct 15, 2020

zuziaka Oct 15, 2020

Choose a reason for hiding this comment

zuziaka Oct 15, 2020

Choose a reason for hiding this comment

codecov-io commented Oct 15, 2020 • edited Loading

Codecov Report

jan-goral Oct 15, 2020

Choose a reason for hiding this comment

zuziaka Oct 16, 2020

Choose a reason for hiding this comment

pawelpasterz commented Oct 19, 2020

piotradamczyk5 left a comment

Choose a reason for hiding this comment

zuziaka commented Oct 6, 2020 •

edited

Loading

zuziaka Oct 6, 2020 •

edited

Loading

codecov-io commented Oct 15, 2020 •

edited

Loading