Race condition in vstream_manager

[deepthi]

Overview of the Issue

unit_test_race now consistently fails in #8696 with this trace

WARNING: DATA RACE
Write at 0x00c0088ed1b0 by goroutine 786:
  vitess.io/vitess/go/vt/vtgate.(*vstream).sendEvents.func4()
      /home/runner/work/vitess/vitess/go/vt/vtgate/vstream_manager.go:273 +0x64
  sync.(*Once).doSlow()
      /opt/hostedtoolcache/go/1.17.0/x64/src/sync/once.go:68 +0x127
  sync.(*Once).Do()
      /opt/hostedtoolcache/go/1.17.0/x64/src/sync/once.go:59 +0x46
  vitess.io/vitess/go/vt/vtgate.(*vstream).sendEvents()
      /home/runner/work/vitess/vitess/go/vt/vtgate/vstream_manager.go:272 +0x6be
  vitess.io/vitess/go/vt/vtgate.(*vstream).stream·dwrap·84()
      /home/runner/work/vitess/vitess/go/vt/vtgate/vstream_manager.go:232 +0x58

Previous read at 0x00c0088ed1b0 by goroutine 865:
  vitess.io/vitess/go/vt/vtgate.(*vstream).sendAll()
      /home/runner/work/vitess/vitess/go/vt/vtgate/vstream_manager.go:595 +0x18f
  vitess.io/vitess/go/vt/vtgate.(*vstream).streamFromTablet.func2()
      /home/runner/work/vitess/vitess/go/vt/vtgate/vstream_manager.go:517 +0x804
  vitess.io/vitess/go/vt/vttablet/sandboxconn.(*SandboxConn).VStream()
      /home/runner/work/vitess/vitess/go/vt/vttablet/sandboxconn/sandboxconn.go:486 +0x668
  vitess.io/vitess/go/vt/vtgate.(*vstream).streamFromTablet()
      /home/runner/work/vitess/vitess/go/vt/vtgate/vstream_manager.go:476 +0x1169
  vitess.io/vitess/go/vt/vtgate.(*vstream).startOneStream.func1()
      /home/runner/work/vitess/vitess/go/vt/vtgate/vstream_manager.go:306 +0xdb

Reproduction Steps

$ cd go/src/vitess.io/vitess/go/vt/vtgate
$ for i in {1..10}; do go test -timeout 2m -race; done

Binary version

Git revision 8b7a4cb branch 'main'

[deepthi]

I attempted to fix this by synchronizing on vs.mu before writing to vs.err but that led to the tests hanging.
A more careful review of the code is required while implementing synchronization.

[deepthi]

Fixed in #8696