[Bug]: ValidateRunnerXlangTest$SingleInputOutputTest.test failing for various x-lang test suites

[chamikaramj]

What happened?

For example,

https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/5756/
https://ci-beam.apache.org/job/beam_PostCommit_XVR_JavaUsingPython_Dataflow/787/
https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/3008/

Stacktrace
java.lang.RuntimeException: expansion service error: Traceback (most recent call last):
File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_XVR_Direct/src/build/gradleenv/1922375555/lib/python3.7/site-packages/apache_beam/runners/portability/expansion_service.py", line 65, in Expand
ptransform.PTransform.from_runner_api(request.transform, context))
File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_XVR_Direct/src/build/gradleenv/1922375555/lib/python3.7/site-packages/apache_beam/transforms/ptransform.py", line 768, in from_runner_api
context)
File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_XVR_Direct/src/build/gradleenv/1922375555/lib/python3.7/site-packages/apache_beam/runners/portability/expansion_service_test.py", line 113, in from_runner_api_parameter
return PrefixTransform(parse_string_payload(payload)['data'])
File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_XVR_Direct/src/build/gradleenv/1922375555/lib/python3.7/site-packages/apache_beam/runners/portability/expansion_service_test.py", line 349, in parse_string_payload
return RowCoder(payload.schema).decode(payload.payload)._asdict()
File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_XVR_Direct/src/build/gradleenv/1922375555/lib/python3.7/site-packages/apache_beam/coders/row_coder.py", line 64, in init
self._type_hint = named_tuple_from_schema(self.schema)
File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_XVR_Direct/src/build/gradleenv/1922375555/lib/python3.7/site-packages/apache_beam/typehints/schemas.py", line 473, in named_tuple_from_schema
schema_registry=schema_registry).named_tuple_from_schema(schema)
File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_XVR_Direct/src/build/gradleenv/1922375555/lib/python3.7/site-packages/apache_beam/typehints/schemas.py", line 432, in named_tuple_from_schema
assert schema.id
AssertionError

Issue Priority

Priority: 1

Issue Component

Component: test-failures

[chamikaramj]

Potential culprit: #22679

[chamikaramj]

@yeandy will you be able to check since @TheNeuralBit is out ? Can we just remove the assertion ?

[yeandy]

I believe we can remove the assertion. In #22679, most of the code around it was simply refactored with non change, except for the addition of that particular line.

I'm guessing the value during these tests is either '' or None, which doesn't actually break the tests. (Is this intended though?)

[chamikaramj]

ID is probably set to empty string which is done here: https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaTranslation.java#L83

According to the proto, the ID has to be set to a UUID: https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema.proto#L40

So the actual fix might have to be done to the SchemaTranslation class (and any other locations where we do not set an ID when generating the Schema proto).

I'll merge #22750 though to fix the tests.

[TheNeuralBit]

Thanks @chamikaramj and @yeandy for addressing this while I was out. IIUC the action here now is to make sure that the Java SDK always generates an ID for schemas (then potentially add the assertion back in Python).

Lowering to P2 since this is now feature work.