ParDoTest cases that should be ValidatesRunner

[kennknowles]

Discovered that these ParDoTest cases were not running because they were annotated only as NeedsRunner.

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	SDK	Dataflow	Flink	Samza	Spark	Twister2
Go		---		---		---
Java
Python				---		---
XLang				---		---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---		---	---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[kennknowles]

run flink validatesrunner

[kennknowles]

run spark validatesrunner

[kennknowles]

run samza validatesrunner

[boyuanzz]

Should we give a clear instruction to dev on when to use NeedsRunner and when to use ValidatesRunner? NeedsRunner suite will be picked with DirectRunner during Java Precommit, which gives an early signal comparing to post commit.
To my understanding, NeedsRunner is for testing new built composite transform, which requires a runner to run. ValidatesRunner is more for validating assumptions that SDK makes on runners.

[kennknowles]

Yes, that is exactly the difference. NeedsRunner is really a hack to break the dependency cycle. We could even rename it to CoreSdkTestToRunOnDirectRunner so we don't have to document (people won't find documentation anyhow).

We can make a stronger statement about ValidatesRunner IMO: the ValidatesRunner tests should be the definition of a correct runner. Of course the runner also calls back into the SDK so it does test both, but mostly the runner. We should ensure that the SDK and tests are correct by using the ULR so that we know that any failure in these tests on other runners indicates a limitation of the runner.

[kennknowles]

Combine is an example of where this difference becomes more ambiguous: it is a composite, but certainly runners will want to run these tests. So for now, when a transform is "very likely to be executed as a runner primitive" we mark it ValidatesRunner.

[kennknowles]

run samza validatesrunner

[boyuanzz]

Thanks!

[iemejia]

It seems we forgot to run the Portable VR suites here and this change make evident some failures so now Flink PVR Streaming is not passing anymore.

[kennknowles]

Yes, it was noticed same day and there is BEAM-12050 and #14333. I forgot to tag you on it, if you want to take a look.

[kennknowles]

(I'm not working today, but I wanted to reply to unblock you if there is any need)

[iemejia]

Thanks for letting me know I will take care there !