Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Difficulty reprocessing failed records #32866

Closed
1 of 17 tasks
ujjwalrajanand opened this issue Oct 18, 2024 · 3 comments
Closed
1 of 17 tasks

Difficulty reprocessing failed records #32866

ujjwalrajanand opened this issue Oct 18, 2024 · 3 comments
Labels

Comments

@ujjwalrajanand
Copy link

What happened?

https://github.com/vaishnavipandey-vp/Apache-Beam-YAML/blob/c467c59f5f13a00514e90be1cfd505982577524b/Apache-Beam-YAML/Pipelines/exceptionHandlingMultiple.yaml#L19C1-L28C35

Not able to process the failed records obtained from the previous transform as it is generated in a JSON format. We are required to extract the element values of the failed records from the JSON file in order to process it further.

image

Issue Failure

Failure: Test is continually failing

Issue Priority

Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam YAML
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Infrastructure
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner
@robertwb robertwb changed the title [Failing Test]: Processing failed records Difficulty reprocessing failed records Oct 18, 2024
@robertwb
Copy link
Contributor

I created #32874 to make the documentation a bit clearer.

@robertwb
Copy link
Contributor

I started a discussion at https://lists.apache.org/thread/jhgsp1rw7l1xr6knpjl9mhvnjwjf3zwf about how to possibly make this easier.

@robertwb
Copy link
Contributor

robertwb commented Dec 6, 2024

This is now resolved with the StripErrorMetadata transform, see also https://beam.apache.org/documentation/sdks/yaml-errors/

@robertwb robertwb closed this as completed Dec 6, 2024
@github-actions github-actions bot added this to the 2.62.0 Release milestone Dec 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants