Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[prism]: unsupported feature "WindowStrategy.OutputTime" set with value (LATEST|EARLIEST)_IN_PANE #31462

Closed
Tracked by #29650
damondouglas opened this issue May 30, 2024 · 2 comments · Fixed by #31806 or #31979
Assignees

Comments

@damondouglas
Copy link
Contributor

damondouglas commented May 30, 2024

Several Java tests fail on Prism with errors:

unsupported feature "WindowStrategy.OutputTime" set with value LATEST_IN_PANE
unsupported feature "WindowStrategy.OutputTime" set with value EARLIEST_IN_PANE

Steps to reproduce:

  1. Run Prism server.
./gradlew :runners:prism:runServer
  1. Execute :runners:portability:java:ulrLoopbackValidatesRunnerTests on Prism:

Many tests fail with these errors, but to see the output of one quickly:

TEST=org.apache.beam.sdk.transforms.ReshuffleTest
./gradlew :runners:portability:java:ulrLoopbackValidatesRunnerTests -PjobEndpoint=localhost:8073 --tests="$TEST"

Parent tracker: #28187
Related: #31438

@lostluck
Copy link
Contributor

Re-opening since the feature isn't fully implemented at this time.

https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/runners/prism/internal/handlerunner.go#L343

Latest and Earliest in Pane must also take into account the rest of the Aggregation, which means using the previous computed time for the aggregation, and not simply the most recently seen timestamp for the element.

https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/runners/prism/internal/handlerunner.go#L343

Assuming the aggregation struct exists kt.time contains the previous value during the aggregation.

https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/runners/prism/internal/handlerunner.go#L350

@lostluck lostluck reopened this Jul 16, 2024
@lostluck lostluck self-assigned this Jul 16, 2024
@lostluck lostluck changed the title [Bug][Prism]: unsupported feature "WindowStrategy.OutputTime" set with value (LATEST|EARLIEST)_IN_PANE [prism]: unsupported feature "WindowStrategy.OutputTime" set with value (LATEST|EARLIEST)_IN_PANE Jul 17, 2024
@lostluck
Copy link
Contributor

Looks like the Go SDK doesn't yet support OutputTime configuration in the windowing strategy:

#20436 tracks allowing and correctly implementing OutputTime SDK side (accounting for CombinerLifitng too).

This will be implemented and then validate via the Java tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment