Skip to content

Commit

Permalink
changefeedccl: Handle node unavailable error.
Browse files Browse the repository at this point in the history
Treat NodeUnavailable error as a retryable changefeed error.

NodeUnavaiable error may be returned by changefeed processors if the
node is being shutdown/drained.  However, this error return is racy.
Sometimes, the coordinator would see "rpc context cancellation" error
instead -- in those cases it would treat the error as retryable.
However, sometimes it is possible to get this error propagated
(for example: when server shutdown races with starting up kv feed,
which uses Stopper, whcih may return NodeUnavailable error if it's
being shutdown).

Release Notes (bug fix): Treat node unavailable error as a retryable
changefeed error.
  • Loading branch information
Yevgeniy Miretskiy committed Jun 13, 2022
1 parent 999a127 commit 9064a0c
Show file tree
Hide file tree
Showing 2 changed files with 5 additions and 1 deletion.
1 change: 1 addition & 0 deletions pkg/ccl/changefeedccl/changefeedbase/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ go_library(
"//pkg/clusterversion",
"//pkg/jobs/joberror",
"//pkg/jobs/jobspb",
"//pkg/roachpb",
"//pkg/settings",
"//pkg/sql",
"//pkg/sql/catalog",
Expand Down
5 changes: 4 additions & 1 deletion pkg/ccl/changefeedccl/changefeedbase/errors.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import (
"strings"

"github.com/cockroachdb/cockroach/pkg/jobs/joberror"
"github.com/cockroachdb/cockroach/pkg/roachpb"
"github.com/cockroachdb/cockroach/pkg/sql/flowinfra"
"github.com/cockroachdb/errors"
)
Expand Down Expand Up @@ -117,7 +118,9 @@ func IsRetryableError(err error) bool {
return true
}

return (joberror.IsDistSQLRetryableError(err) || flowinfra.IsNoInboundStreamConnectionError(err))
return (joberror.IsDistSQLRetryableError(err) ||
flowinfra.IsNoInboundStreamConnectionError(err) ||
errors.HasType(err, (*roachpb.NodeUnavailableError)(nil)))
}

// MaybeStripRetryableErrorMarker performs some minimal attempt to clean the
Expand Down

0 comments on commit 9064a0c

Please sign in to comment.