Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug][prism]: Support receiving Java's CoGBKList Result #28544

Open
1 of 15 tasks
lostluck opened this issue Sep 19, 2023 · 2 comments
Open
1 of 15 tasks

[Bug][prism]: Support receiving Java's CoGBKList Result #28544

lostluck opened this issue Sep 19, 2023 · 2 comments

Comments

@lostluck
Copy link
Contributor

lostluck commented Sep 19, 2023

What happened?

Run the TestXlang_CoGroupBy test from the Go validates runner suite.

Data gets corrupted, and if the Go SDK does loopback execution the passert returns the following:

        actual PCollection does not match expected values
        =========
        0 correct entries (present in both)
        =========
        2 unexpected entries (present in actual, missing in expected)
        +++
        1964988:[]
        +++
        1:[]
        =========
        2 missing entries (missing in actual, present in expected)
        ---
        0:[1 2 4]
        ---
        1:[3 5 6]

As this is only happening in prism (other runners seem to cope), either prism's unfused execution is causing corruption on the Java side, OR it's prism receiving it is causing the corruption. Either way, it needs to be fixed so the pipeline passes.

The test in question is here:

func TestXLang_CoGroupBy(t *testing.T) {

It needs to be unfilered from the prism run here though:

"TestXLang_CoGroupBy",

The following action workflow runs the suite:

https://github.com/apache/beam/actions/workflows/beam_PreCommit_GoPrism.yml

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner
@lostluck
Copy link
Contributor Author

lostluck commented Nov 7, 2023

Fusion was added in #29306 which is currently enabled by default. It's most likely that fusion will be enabled by default for xlang based pipelines anyway (if not for non-go pipelines). This issue can be closed once the CoGBK test is no longer filtered out.

@lostluck
Copy link
Contributor Author

lostluck commented Nov 7, 2023

cc: @robertwb

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants