Tests run in error when using both notPackage and notClass test targets

[jschear]

When using this config:

test-targets:
  - notPackage foo
  - notClass whatever.Bar

I would expect that all tests would run, except for those in the foo package and the class whatever.Bar. This is the behavior when those arguments are passed to AndroidJUnitTestRunner. But it seems that tests that match any of the filters are included in the run, so all tests are actually included in the run.

Using notTestFile is a workaround:

test-targets:
  - notTestFile excludeTests.txt

$ cat excludeTests.txt
foo
whatever.Bar

These tests for TestFiltersTest.kt highlight the discrepancy:

    @Test
    fun testFilteringClassAndPackageNegative() {
        val filter = fromTestTargets(listOf("notPackage foo", "notClass whatever.Bar"))

        assertThat(filter.shouldRun(FOO_PACKAGE)).isFalse()
        assertThat(filter.shouldRun(BAR_CLASSNAME)).isFalse()
        assertThat(filter.shouldRun(BAR_PACKAGE)).isTrue()
    }

    @Test
    fun testFilteringClassAndPackageNegativeFromFile() {
        val file = TestHelper.getPath(TEST_FILE_2) // contents: foo whatever.Bar
        val filePath = file.toString()

        val filter = fromTestTargets(listOf("notTestFile $filePath"))

        assertThat(filter.shouldRun(FOO_PACKAGE)).isFalse()
        assertThat(filter.shouldRun(BAR_CLASSNAME)).isFalse()
        assertThat(filter.shouldRun(BAR_PACKAGE)).isTrue()
    }

[bootstraponline]

Thanks for reporting the bug!

[jan-goral]

I have debugged both tests mentioned above.
Below I paste descriptions of generated test filters.

• testFilteringClassAndPackageNegative:
  allOf [anyOf [not withPackageName foo, not withClassName whatever.Bar]]
• testFilteringClassAndPackageNegativeFromFile:
  allOf [anyOf [not withPackageName foo, whatever.Bar]]

By the way, descriptions are misleading without additional brackets and should be read as below.

• allOf [anyOf [not (withPackageName (foo)), not (withClassName (whatever.Bar))]]
• allOf [anyOf [not (withPackageName (foo, whatever.Bar))]]

Any way the second case works well. But
the first one fails because
allOf [anyOf [not withPackageName foo, not withClassName whatever.Bar]] for foo.ClassName#testName or whatever.Bar#testName gives true, because anyOf(false, true) gives true. So we cannot wrap two or more exclusions into anyOf.

I have an idea how to fix that, we need to separate includes and excludes filters, only the first one can be wrapped into anyOf. The pull request will coming soon.

[bootstraponline]

Should we also try to update the descriptions so that's clear when debugging?

[jan-goral]

Should we also try to update the descriptions so that's clear when debugging?

Yes, of course.