WIP - remote build: fix streaming and error handling

[vrothberg]

Address a number of issues in the streaming logic in remote build, most
importantly an error in using buffered channels on the server side.

The pattern below does not guarantee that the channel is entirely read
before the context fires.

for {
        select {
                case <- bufferedChannel:
                ...
                case <- ctx.Done():
                ...
        }
}

Fixes: #10154
Signed-off-by: Valentin Rothberg [email protected]

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vrothberg

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [vrothberg]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[vrothberg]

@edsantiago, it looks like we were masking errors before. I think this will help but it's hard to reproduce.

[edsantiago]

"Hard to reproduce"? Hold my beer. I rebased #9887 on top of this PR and ran it through CI. It failed. Looks like exactly the same failure pattern, except the error message is now "server probably quit" instead of "stream dropped".

[vrothberg]

"Hard to reproduce"? Hold my beer. I rebased #9887 on top of this PR and ran it through CI. It failed. Looks like exactly the same failure pattern, except the error message is now "server probably quit" instead of "stream dropped".

Thanks, Ed! I will give it another shot tomorrow.

[vrothberg]

@edsantiago one round without flakes. Let's try another time.

I noticed that we need to distinguish between EOF and unexpected EOF. Not yet sure if that's the key or not.

[edsantiago]

Ah, you rebased on my buildah-bud PR, clever. Why do you say "without flakes" though? I see the same old same old

[vrothberg]

Ah, you rebased on my buildah-bud PR, clever. Why do you say "without flakes" though? I see the same old same old

I think this one happened in "another time". First run went fine. The hunt continues...

[vrothberg]

Ed will cherry-pick my commit here into his PR, so I am closing.