[Task]: Pipeline testing framework

[laysakura]

What needs to happen?

We already hold some transforms to make assertion like:

beam/sdks/rust/tests/primitives_test.rs

Line 57 in 54b4d23

.apply(AssertEqualUnordered::new(&[1]))

But we might lack some test transforms compared to other SDKs, or rather we might be able to develop more sophisticated test framework to test pipelines.

The task should be like:

1. First, conduct survey on other SDKs and write a design document (in google docs and add to the wiki page) to make aligned image of the next test framework of Rust SDK.
2. Request a review from other contributors ( @laysakura and/or @sjvanrossum would be happy to review it).
3. Develop it.

Issue Priority

Priority: 3 (nice-to-have improvement)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[dahlbaek]

I skimmed over what the different sdks provide, and personally found the following to seem quite useful

• apply a predicate function to each input element and fail if the predicate function returns false:
  • go https://github.com/apache/beam/blob/f1adfdd7f24b65742bf66292943feba1a1ec94df/sdks/go/pkg/beam/testing/passert/equals.go#L67-L105
  • java https://github.com/apache/beam/blob/f1adfdd7f24b65742bf66292943feba1a1ec94df/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java#L595-L607
  • typescript https://github.com/apache/beam/blob/f1adfdd7f24b65742bf66292943feba1a1ec94df/sdks/typescript/src/apache_beam/testing/assert.ts#L69-L108
• count number of elements in a collection:
  • https://github.com/apache/beam/blob/f1adfdd7f24b65742bf66292943feba1a1ec94df/sdks/go/pkg/beam/testing/passert/count.go
• materialize pipeline with no intention of running it:
  • https://github.com/apache/beam/blob/f1adfdd7f24b65742bf66292943feba1a1ec94df/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/CrashingRunner.java
• component specifically for testing combinefns:
  • https://github.com/apache/beam/blob/f1adfdd7f24b65742bf66292943feba1a1ec94df/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/CombineFnTester.java
• test pipeline to simplify instantiation of test pipeline to be run against different runners:
  • https://github.com/apache/beam/blob/f2240ffd9c57a16266b240a116e93ed9a877e623/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java
  • https://github.com/apache/beam/blob/f2240ffd9c57a16266b240a116e93ed9a877e623/sdks/python/apache_beam/testing/test_pipeline.py

On top of those, we will probably want transforms crafted specifically to validate streaming pipelines, but it might not make much sense to attempt to implement such validations before we implement proper window/watermark handling. finally, I wonder if it would be useful with a collection factory that shuffles the provided input for tests rather than emitting the values in a deterministic order 🤔

[laysakura]

@dahlbaek Thank you for your wholistic investigation.

Things I feel really useful

• apply a predicate function to each input element and fail if the predicate function returns false
• count number of elements in a collection
• component specifically for testing combinefns
• a collection factory that shuffles the provided input for tests rather than emitting the values in a deterministic order

Things I feel interesting (nice to have)

• materialize pipeline with no intention of running it
• test pipeline to simplify instantiation of test pipeline to be run against different runners

Additionally, I will totally agree with the following opinion.

we will probably want transforms crafted specifically to validate streaming pipelines, but it might not make much sense to attempt to implement such validations before we implement proper window/watermark handling.

Could you start to write a design doc?