You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The Go SDK currently retains an arbitrary timestamp per key per bundle when performing a lifted combine.
However, depending on the windowing strategy, a prefered time could be specified.
At present this implementation is "correct", as the default output time is Unspecified, and there's no user mechanism to configure a windowing strategy to this granularity.
So there are a few parts to this.
Propagate the windowing strategy information to exec.LiftedCombine somehow and implement the correct output. This can be done whether or not 2 is implemented.
Provide a trigger configuration for beam.WindowInto, so this can be configured on the user side. This is significantly more work.
This matters only when using windows that are not the Global Window, and when using a Lifted Combine, which commonly only happens in batch contexts. However, since Beam is a unified model, the windowing features should work correctly in both execution modes of a Go SDK pipeline.
Imported from Jira BEAM-10302. Original Jira may contain additional context.
Reported by: lostluck.
The text was updated successfully, but these errors were encountered:
The Go SDK currently retains an arbitrary timestamp per key per bundle when performing a lifted combine.
However, depending on the windowing strategy, a prefered time could be specified.
beam/model/pipeline/src/main/proto/beam_runner_api.proto
Line 901 in a5b2046
The code in question for the Go SDK:
https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/combine.go#L395
At present this implementation is "correct", as the default output time is Unspecified, and there's no user mechanism to configure a windowing strategy to this granularity.
So there are a few parts to this.
This matters only when using windows that are not the Global Window, and when using a Lifted Combine, which commonly only happens in batch contexts. However, since Beam is a unified model, the windowing features should work correctly in both execution modes of a Go SDK pipeline.
Imported from Jira BEAM-10302. Original Jira may contain additional context.
Reported by: lostluck.
The text was updated successfully, but these errors were encountered: