Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Python PreCommit dependency compatibility test not run or redundant setup #25115

Merged
merged 2 commits into from
Jan 23, 2023
Merged

Fix Python PreCommit dependency compatibility test not run or redundant setup #25115

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
010d1d3
Fix Python PreCommit dependency compatibility test not run or redunda…
Abacn Jan 23, 2023
9fc7ef0
Also move dataframe comp tests to preCommitPyCoverage
Abacn Jan 23, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .test-infra/jenkins/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ Beam Jenkins overview page: [link](https://ci-beam.apache.org/)
| beam_PreCommit_Python_Examples | [commit](https://ci-beam.apache.org/job/beam_PreCommit_Python_Examples_Commit/), [cron](https://ci-beam.apache.org/job/beam_PreCommit_Python_Examples_Cron/), [phrase](https://ci-beam.apache.org/job/beam_PreCommit_Python_Examples_Phrase/) | `Run Python_Examples PreCommit` | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python_Examples_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python_Examples_Cron) |
| beam_PreCommit_Python_Runners | [commit](https://ci-beam.apache.org/job/beam_PreCommit_Python_Runners_Commit/), [cron](https://ci-beam.apache.org/job/beam_PreCommit_Python_Runners_Cron/), [phrase](https://ci-beam.apache.org/job/beam_PreCommit_Python_Runners_Phrase/) | `Run Python_Runners PreCommit` | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python_Runners_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python_Runners_Cron) |
| beam_PreCommit_Python_Transforms | [commit](https://ci-beam.apache.org/job/beam_PreCommit_Python_Transforms_Commit/), [cron](https://ci-beam.apache.org/job/beam_PreCommit_Python_Transforms_Cron/), [phrase](https://ci-beam.apache.org/job/beam_PreCommit_Python_Transforms_Phrase/) | `Run Python_Transforms PreCommit` | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python_Transforms_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python_Transforms_Cron) |
| beam_PreCommit_Python_Coverage | [commit](https://ci-beam.apache.org/job/beam_PreCommit_Python_Coverage_Commit/), [cron](https://ci-beam.apache.org/job/beam_PreCommit_Python_Coverage_Cron/), [phrase](https://ci-beam.apache.org/job/beam_PreCommit_Python_Coverage_Phrase/) | `Run Python_Coverage PreCommit` | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python_Coverage_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python_Coverage_Cron) |
| beam_PreCommit_PythonDocker | [commit](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Commit/), [cron](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/), [phrase](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Phrase/) | `Run PythonDocker PreCommit` | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/) |
| beam_PreCommit_PythonDocs| [commit](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Commit/), [cron](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/), [phrase](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Phrase/) | `Run PythonDocs PreCommit` | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/) |
| beam_PreCommit_Python_PVR_Flink | [commit](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Commit/), [cron](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/), [phrase](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Phrase/) | `Run Python_PVR_Flink PreCommit` | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron) |
Expand Down
81 changes: 39 additions & 42 deletions sdks/python/test-suites/tox/py38/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,82 +36,83 @@ testPy38Cython.mustRunAfter testPython38, testPy38CloudCoverage

// TODO(BEAM-12000): Move tasks that aren't specific to 3.8 to Py 3.9.
def posargs = project.findProperty("posargs") ?: ""
def runDataframesTests = (posargs == "" || posargs.contains("dataframes"))
logger.info('Running dataframes tests: ' + runDataframesTests)

// PyCoverage Precommit runs test suites that evaluate test coverage and compatibility of
// particular dependencies. It is exercised on a single Python version.
project.tasks.register("preCommitPyCoverage") {
dependsOn = ["testPy38CloudCoverage"]
}

// Create a test task for each major version of pyarrow
toxTask "testPy38pyarrow-0", "py38-pyarrow-0", "${posargs}"
test.dependsOn "testPy38pyarrow-0"
preCommitPy38.dependsOn "testPy38pyarrow-0"
preCommitPyCoverage.dependsOn "testPy38pyarrow-0"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the right move for these tests, we should probably actually do the same thing for the dataframes tests so that they get counted towards codecoverage.

I'm approving this PR, feel free to merge as is or make that change. If you merge as is, I'll put up a follow up PR to move the dataframes tests into the coverage job.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, moving dataframe compatibility tests into preCommitPyCoverage


toxTask "testPy38pyarrow-1", "py38-pyarrow-1", "${posargs}"
test.dependsOn "testPy38pyarrow-1"
preCommitPy38.dependsOn "testPy38pyarrow-1"
preCommitPyCoverage.dependsOn "testPy38pyarrow-1"

toxTask "testPy38pyarrow-2", "py38-pyarrow-2", "${posargs}"
test.dependsOn "testPy38pyarrow-2"
preCommitPy38.dependsOn "testPy38pyarrow-2"
preCommitPyCoverage.dependsOn "testPy38pyarrow-2"

toxTask "testPy38pyarrow-3", "py38-pyarrow-3", "${posargs}"
test.dependsOn "testPy38pyarrow-3"
preCommitPy38.dependsOn "testPy38pyarrow-3"
preCommitPyCoverage.dependsOn "testPy38pyarrow-3"

toxTask "testPy38pyarrow-4", "py38-pyarrow-4", "${posargs}"
test.dependsOn "testPy38pyarrow-4"
preCommitPy38.dependsOn "testPy38pyarrow-4"
preCommitPyCoverage.dependsOn "testPy38pyarrow-4"

toxTask "testPy38pyarrow-5", "py38-pyarrow-5", "${posargs}"
test.dependsOn "testPy38pyarrow-5"
preCommitPy38.dependsOn "testPy38pyarrow-5"
preCommitPyCoverage.dependsOn "testPy38pyarrow-5"

toxTask "testPy38pyarrow-6", "py38-pyarrow-6", "${posargs}"
test.dependsOn "testPy38pyarrow-6"
preCommitPy38.dependsOn "testPy38pyarrow-6"
preCommitPyCoverage.dependsOn "testPy38pyarrow-6"

toxTask "testPy38pyarrow-7", "py38-pyarrow-7", "${posargs}"
test.dependsOn "testPy38pyarrow-7"
preCommitPy38.dependsOn "testPy38pyarrow-7"
preCommitPyCoverage.dependsOn "testPy38pyarrow-7"

toxTask "testPy38pyarrow-8", "py38-pyarrow-8", "${posargs}"
test.dependsOn "testPy38pyarrow-8"
preCommitPy38.dependsOn "testPy38pyarrow-8"
preCommitPyCoverage.dependsOn "testPy38pyarrow-8"

toxTask "testPy38pyarrow-9", "py38-pyarrow-9", "${posargs}"
test.dependsOn "testPy38pyarrow-9"
preCommitPy38.dependsOn "testPy38pyarrow-9"

// Dataframes tests won't get auto-filtered by path since they explicitly only run on the dataframes directory in tox.ini
if (runDataframesTests) {
// Create a test task for each minor version of pandas
toxTask "testPy38pandas-11", "py38-pandas-11", "${posargs}"
test.dependsOn "testPy38pandas-11"
preCommitPy38.dependsOn "testPy38pandas-11"

toxTask "testPy38pandas-12", "py38-pandas-12", "${posargs}"
test.dependsOn "testPy38pandas-12"
preCommitPy38.dependsOn "testPy38pandas-12"

toxTask "testPy38pandas-13", "py38-pandas-13", "${posargs}"
test.dependsOn "testPy38pandas-13"
preCommitPy38.dependsOn "testPy38pandas-13"

toxTask "testPy38pandas-14", "py38-pandas-14", "${posargs}"
test.dependsOn "testPy38pandas-14"
preCommitPy38.dependsOn "testPy38pandas-14"

toxTask "testPy38pandas-15", "py38-pandas-15", "${posargs}"
test.dependsOn "testPy38pandas-15"
preCommitPy38.dependsOn "testPy38pandas-15"
}
preCommitPyCoverage.dependsOn "testPy38pyarrow-9"

// Create a test task for each minor version of pandas
toxTask "testPy38pandas-11", "py38-pandas-11", "${posargs}"
test.dependsOn "testPy38pandas-11"
preCommitPyCoverage.dependsOn "testPy38pandas-11"

toxTask "testPy38pandas-12", "py38-pandas-12", "${posargs}"
test.dependsOn "testPy38pandas-12"
preCommitPyCoverage.dependsOn "testPy38pandas-12"

toxTask "testPy38pandas-13", "py38-pandas-13", "${posargs}"
test.dependsOn "testPy38pandas-13"
preCommitPyCoverage.dependsOn "testPy38pandas-13"

toxTask "testPy38pandas-14", "py38-pandas-14", "${posargs}"
test.dependsOn "testPy38pandas-14"
preCommitPyCoverage.dependsOn "testPy38pandas-14"

toxTask "testPy38pandas-15", "py38-pandas-15", "${posargs}"
test.dependsOn "testPy38pandas-15"
preCommitPyCoverage.dependsOn "testPy38pandas-15"

// Create a test task for each minor version of pytorch
toxTask "testPy38pytorch-19", "py38-pytorch-19", "${posargs}"
test.dependsOn "testPy38pytorch-19"
preCommitPy38.dependsOn "testPy38pytorch-19"
preCommitPyCoverage.dependsOn "testPy38pytorch-19"

toxTask "testPy38pytorch-110", "py38-pytorch-110", "${posargs}"
test.dependsOn "testPy38pytorch-110"
preCommitPy38.dependsOn "testPy38pytorch-110"
preCommitPyCoverage.dependsOn "testPy38pytorch-110"

toxTask "whitespacelint", "whitespacelint", "${posargs}"

Expand All @@ -126,10 +127,6 @@ task archiveFilesToLint(type: Zip) {
}
}

project.tasks.register("preCommitPyCoverage") {
dependsOn = ["testPy38CloudCoverage"]
}

task unpackFilesToLint(type: Copy) {
from zipTree("$buildDir/dist/files-to-whitespacelint.zip")
into "$buildDir/files-to-whitespacelint"
Expand Down