Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: textio and parquetio are not -beam_strict compatible #26468

Closed
1 of 15 tasks
jeremyje opened this issue Apr 28, 2023 · 0 comments · Fixed by #26469
Closed
1 of 15 tasks

[Bug]: textio and parquetio are not -beam_strict compatible #26468

jeremyje opened this issue Apr 28, 2023 · 0 comments · Fixed by #26469
Assignees

Comments

@jeremyje
Copy link
Contributor

What happened?

If you run a pipeline with -beam_strict in Go you'll get the following errors from the v2 SDK.

There are possibly more issues like this within the SDK as well.

Full error:
strictness check failed
        caused by:
        validating pipeline with vet runner
pipeline is not performant, see diagnostic summary:
/**
PTransform Audit:
pardo github.com/apache/beam/sdks/v2/go/pkg/beam/io/textio.writeFileFn struct[[*textio.writeFileFn]] FOUND textio.writeFileFn
        - ProcessElementFn function[[func(context.Context, int, func(*string) bool) error]] NEED_SHIM OK
pardo github.com/apache/beam/sdks/v2/go/pkg/beam/io/parquetio.parquetWriteFn struct[[*parquetio.parquetWriteFn]] FOUND parquetio.parquetWriteFn
        - ProcessElementFn function[[func(context.Context, int, func(*typex.X) bool) error]] NEED_SHIM OK

Summary
All exported?: true
0        Imports
0        Functions
0        Types
2        Shims
0        Emits
0        Inputs
Pipeline is not performant:
        requires additional shim generation
        Good News! All identifiers are exported; the pipeline's package can be patched with generated output.
Diagnostic output pre-amble for the code generator
Functions
Types
Shims
func(context.Context, int, func(*string) bool) error, func(context.Context, int, func(*string) bool) error
func(context.Context, int, func(*typex.X) bool) error, func(context.Context, int, func(*typex.X) bool) error
Emitters
Iterators 
*/

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner
@github-actions github-actions bot added this to the 2.48.0 Release milestone Apr 28, 2023
@lostluck lostluck changed the title [Bug]: fileio and parquetio are not -beam_strict compatible [Bug]: textio and parquetio are not -beam_strict compatible Apr 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant