Backport of alloc exec: fix panics after stream close into release/1.5.x

[hc-github-team-nomad-core]

Backport

This PR is auto-generated from #19932 to be assessed for backporting due to the inclusion of the label backport/1.5.x.

🚨

Warning automatic cherry-pick of commits failed. If the first commit failed,
you will see a blank no-op commit below. If at least one commit succeeded, you
will see the cherry-picked commits up to, not including, the commit where
the merge conflict occurred.

The person who merged in the original PR is:
@tgross
This person should manually cherry-pick the original PR into a new backport PR,
and close this one when the manual backport PR is merged in.

merge conflict error: POST https://api.github.com/repos/hashicorp/nomad/merges: 409 Merge conflict []

The below text is copied from the body of the original PR.

---

In #19172 we added a check on websocket errors to see if they were one of several benign "close" messages. This change inadvertently assumed that other messages used for close would not implement HTTPCodedError. When errors like the following are received:

msgpack decode error [pos 0]: io: read/write on closed pipe"

they are sent from the inner loop as though they were a "real" error, but the channel is already being closed with a "close" message.

This allowed many more attempts to pass thru a previously-undiscovered race condition in the two goroutines that stream RPC responses to the websocket. When the input stream returns an error for any reason (for example, the command we're executing has exited), it will unblock the "outer" goroutine and cause a write to the websocket. If we're concurrently writing the "close error" discussed above, this results in a panic from the websocket library.

This changeset includes two fixes:

• Catch "closed pipe" error correctly so that we're not sending unnecessary error messages.
• Move all writes to the websocket into the same response streaming goroutine. The main handler goroutine will block on a results channel, and the response streaming goroutine will send on that channel with the final error when it's done so it can be reported to the user.

Fixes: #19506

---

In addition to the new unit test and associated websocket test infrastructure, I did some soak testing:

$ for i in {1..200}; do nomad alloc exec 9ee04c03 echo -n "."; done
........................................................................................................................................................................................................%

---

Overview of commits

• e986c29

[hashicorp-cla]

All committers have signed the CLA.