Merge #39466

39466: distsqlrun: log on distsqlrun.outbox Dial error r=yuzefovich a=asubiotto

It's useful to know when an outbox was unable to dial a node in a flow
since this failure could lead to timeouts.

Release note: None

Co-authored-by: Alfonso Subiotto Marqués <[email protected]>

pkg/sql/distsqlrun/outbox.go

@@ -219,6 +219,15 @@ func (m *outbox) mainLoop(ctx context.Context) error {
 		var err error
 		conn, err = m.flowCtx.Cfg.NodeDialer.Dial(ctx, m.nodeID)
 		if err != nil {
+			// Log any Dial errors. This does not have a verbosity check due to being
+			// a critical part of query execution: if this step doesn't work, the
+			// receiving side might end up hanging or timing out.
+			// TODO(asubiotto): On top of ignoring the circuit breaker here (#38602),
+			//  we should also retry a failed Dial. Both changes rest on the argument
+			//  that the gateway planned this query with the assumption that the
+			//  remote node was reachable, the outbox should at least try a bit harder
+			//  to make sure that this is in fact not the case.
+			log.Infof(ctx, "outbox: connection dial error: %+v", err)
 			return err
 		}
 		client := distsqlpb.NewDistSQLClient(conn)