[Failing Test]: :sdks:go:test:ulrValidatesRunner appears to be very flaky at master

[tvalentyn]

What happened?

Sample failure runs: https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/2996

Also failed on some in-flight PRs.

13:19:29 --- PASS: TestEmitParDoAfterGBK (4.50s)
13:19:29 PASS
13:19:29 ok  	github.com/apache/beam/sdks/v2/go/test/regression	124.686s
13:19:29 FAIL
13:19:29 
13:19:29 > Task :sdks:go:test:ulrValidatesRunner FAILED
13:19:29 
13:19:29 FAILURE: Build failed with an exception.
13:19:29 
13:19:29 * Where:
13:19:29 Build file '/home/jenkins/jenkins-slave/workspace/beam_PreCommit_GoPortable_Cron/src/sdks/go/test/build.gradle' line: 152
13:19:29 
13:19:29 * What went wrong:
13:19:29 Execution failed for task ':sdks:go:test:ulrValidatesRunner'.
13:19:29 > Process 'command 'sh'' finished with non-zero exit value 1
13:19:29 
13:19:29 * Try:
13:19:29 > Run with --stacktrace option to get the stack trace.
13:19:29 > Run with --info or --debug option to get more log output.

Issue Failure

Failure: Test is flaky

Issue Priority

Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[tvalentyn]

cc: @lostluck

[lostluck]

Note that this is happening on the Python Portable Runner, but not on any of the other validates runner suites, so it doesn't seem like it's Go side.

I was chasing this down from a different series. It's failing on artifact upload, which doesn't have any recent work on the Go side (https://github.com/apache/beam/tree/master/sdks/go/pkg/beam/artifact).

I'd be concerned it was the datalayer rewrite (#25982), but that wasn't merged until the 28th, and this started happening on the 25th. (The datalayer rewrite does have a flake, but it's in a unit test #26057, not a runner test).

But python's portable runner doesn't seem to have anything new there either.

Very confusing. But then I'd expect to see non-infra flakes in the other runner suite tests.

[lostluck]

2023/03/31 20:19:00 Prepared job with id: go-testxlang_multi-841-7e5f7f7d-e29d-41a5-85b4-c701695d54d4 and staging token: go-testxlang_multi-841-7e5f7f7d-e29d-41a5-85b4-c701695d54d4
    ptest.go:108: Failed to execute job: 	staging artifacts
        failed to stage /tmp/worker-7-1680293938316973953 in 3 attempts: failed to send chunks for /tmp/worker-7-1680293938316973953
        	caused by:
        chunk send failed
        	caused by:
        EOF; failed to send chunks for /tmp/worker-7-1680293938316973953
        	caused by:
        chunk send failed
        	caused by:
        EOF; failed to send chunks for /tmp/worker-7-1680293938316973953
        	caused by:
        chunk send failed
        	caused by:
        EOF; failed to send chunks for /tmp/worker-7-1680293938316973953
        	caused by:
        chunk send failed
        	caused by:
        EOF
--- FAIL: TestXLang_Multi (12.99s)

Same thing for some of the others. But it's very odd that it's happening to the Python runner and not Flink/Spark/Samza.

Feels sort of like a grpc thing, but again, not sure why it's only started recently.

[lostluck]

That path isn't doing the correct thing WRT the error on Send. The EOF means close the stream and see what the server is returning. Typically the EOF means that the server side closed for some reason.
The upload is retrying 3 times and still getting failures like that, so lets try following the proper protocol WRT errors on sends...

[lostluck]

Well that's unexpected:

 ptest.go:108: Failed to execute job: 	staging artifacts
        failed to stage /tmp/worker-5-1680307335622714242 in 3 attempts: failed to send chunks for /tmp/worker-5-1680307335622714242; close error: rpc error: code = Unimplemented desc = Method not found!
        	caused by:
        EOF; failed to send chunks for /tmp/worker-5-1680307335622714242; close error: rpc error: code = Unimplemented desc = Method not found!
        	caused by:
        EOF; failed to send chunks for /tmp/worker-5-1680307335622714242; close error: rpc error: code = Unimplemented desc = Method not found!
        	caused by:
        EOF; failed to send chunks for /tmp/worker-5-1680307335622714242; close error: rpc error: code = Unimplemented desc = Method not found!
        	caused by:
        EOF

So, the Server doesn't seem to have it implemented when the SDK connects? Very strange.
https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Phrase/176/consoleText

[lostluck]

That's because if the "Portable" artifact upload fails, we don't log any of those errors, we just log that the old legacy method doesn't exists. Adding in that logging to see why it's a failing...

[lostluck]

I'm not having any luck replicating that failure right now. Always after debugging is added. I suspect it's related to Jenkins Machine load, so this will have to wait until next week.