Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Pub/Sub sources incompatible with Python MapToFields #31434

Closed
2 of 16 tasks
Polber opened this issue May 29, 2024 · 1 comment
Closed
2 of 16 tasks

[Bug]: Pub/Sub sources incompatible with Python MapToFields #31434

Polber opened this issue May 29, 2024 · 1 comment

Comments

@Polber
Copy link
Contributor

Polber commented May 29, 2024

What happened?

ReadFromPubSub is incompatible with Python MapToFields (or at least I cannot see how to resolve the type errors). A minimal example:

pipeline:
  transforms:
    - type: ReadFromPubSub
      name: Source
      config:
        topic: projects/${PROJECT}/topics/${TOPIC_NAME}
        format: json
        schema:
          type: object
          properties:
            foo: {type: string}
    - type: MapToFields
      name: Transform
      config:
        language: python
        append: true
        fields:
          bar:
            expression: "'baz'"
            output_type: string
      input: Source
    - type: WriteToBigQuery
      name: Sink
      config:
        table: ${PROJECT}:${DATASET}.${TABLE}
        numStreams: 1
      input: Transform

This fails with

Failed to decode Schema due to an error decoding Field proto:

name: "foo"
type {
  nullable: true
  logical_type {
    urn: "beam:logical:pythonsdk_any:v1"
  }
}

despite the fact that foo's type is provided in the Pub/Sub schema

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam YAML
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner
@Polber
Copy link
Contributor Author

Polber commented May 29, 2024

.take-issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants