Nil pointer dereference in tabletserver VStreamRows code path

[mvh-stripe]

Overview of the Issue

We hit a nil pointer dereference when trying to do a shard-split (see below for logs + stack trace).

Here's what I think is happening:

• In the source shard, there's a _vt_PURGE temp table which seems to get copied over to the target shards during schema copy.
• When the target shards issue a VStreamRows request to the source shard for this table, it ends up calling into localVSchema.FindColVindex(tablename). But since this temp table has no vschema associated with it, localVSchema.findTable() returns (nil, nil) and this results in a panic. See https://github.com/vitessio/vitess/blob/main/go/vt/vttablet/tabletserver/vstreamer/local_vschema.go#L36

[2022-01-27 20:54:10.563457] I0127 20:54:10.563318 2997134 engine.go:237] Streaming rows for query select * from _vt_PURGE_7decd7b1794811eca1f506784fed54f7_20220120165616 where in_keyrange('1780-1788'), lastpk: []
[2022-01-27 20:54:10.563655] I0127 20:54:10.563485 2997134 local_vschema.go:76] found internal table _vt_PURGE_7decd7b1794811eca1f506784fed54f7_20220120165616, ignoring in local vschema search

stream ID: 1, tabletAlias: northwest-0000107249. vttablet: rpc error: code = Unknown desc = uncaught panic: runtime error: invalid memory address or nil pointer dereference
. Stack-trace:
runtime/panic.go:212 (0x43619a)
runtime/signal_unix.go:734 (0x450092)
vitess.io/vitess/go/vt/vtgate/vindexes/vschema.go:634 (0xde1c89)
vitess.io/vitess/go/vt/vttablet/tabletserver/vstreamer/local_vschema.go:41 (0x135f684)
vitess.io/vitess/go/vt/vttablet/tabletserver/vstreamer/planbuilder.go:662 (0x1365fe4)
vitess.io/vitess/go/vt/vttablet/tabletserver/vstreamer/planbuilder.go:522 (0x1363a5a)
vitess.io/vitess/go/vt/vttablet/tabletserver/vstreamer/planbuilder.go:417 (0x136273e)
vitess.io/vitess/go/vt/vttablet/tabletserver/vstreamer/rowstreamer.go:135 (0x1367ba4)
vitess.io/vitess/go/vt/vttablet/tabletserver/vstreamer/rowstreamer.go:102 (0x13678b3)
vitess.io/vitess/go/vt/vttablet/tabletserver/vstreamer/engine.go:269 (0x135ee9b)
vitess.io/vitess/go/vt/vttablet/tabletserver/tabletserver.go:1127 (0x146c330)

We are running a slightly older version (checkout commit-SHA 92beb23) if you want the line numbers in traceback to align.

Potential fix:
I'm not sure why the internal/temp tables like _vt_PURGE are even being copied by vtctl/resharder? We should exclude the internal tables in resharder.copySchema (see https://github.com/vitessio/vitess/blob/main/go/vt/wrangler/resharder.go#L294)?

[mattlord]

Related to: #8992

Looks like there's additional work to do in order for these internal tables to be fully ignored.

[mattlord]

I think we need to check for the nil pointer and ignore the table here:

diff --git a/go/vt/vtgate/vindexes/vschema.go b/go/vt/vtgate/vindexes/vschema.go
index 663418b886..186f9920a4 100644
--- a/go/vt/vtgate/vindexes/vschema.go
+++ b/go/vt/vtgate/vindexes/vschema.go
@@ -715,7 +715,7 @@ func ChooseVindexForType(typ querypb.Type) (string, error) {

 // FindBestColVindex finds the best ColumnVindex for VReplication.
 func FindBestColVindex(table *Table) (*ColumnVindex, error) {
-       if len(table.ColumnVindexes) == 0 {
+       if table.ColumnVindexes == nil || len(table.ColumnVindexes) == 0 {
                return nil, fmt.Errorf("table %s has no vindex", table.Name.String())
        }
        var result *ColumnVindex
diff --git a/go/vt/vttablet/tabletserver/vstreamer/uvstreamer.go b/go/vt/vttablet/tabletserver/vstreamer/uvstreamer.go
index 46af517ec2..13abbde96d 100644
--- a/go/vt/vttablet/tabletserver/vstreamer/uvstreamer.go
+++ b/go/vt/vttablet/tabletserver/vstreamer/uvstreamer.go
@@ -27,6 +27,7 @@ import (
        "time"

        vtrpcpb "vitess.io/vitess/go/vt/proto/vtrpc"
+       vtschema "vitess.io/vitess/go/vt/schema"

        "vitess.io/vitess/go/mysql"
        "vitess.io/vitess/go/vt/dbconfigs"
@@ -144,6 +145,10 @@ func (uvs *uvstreamer) buildTablePlan() error {
                }
        }
        for tableName := range tables {
+               if vtschema.IsInternalOperationTableName(tableName) {
+                       log.Infof("Found internal table: %s, not copying in vreplication", tableName)
+                       continue
+               }
                rule, err := matchTable(tableName, uvs.filter, tables)
                if err != nil {
                        return err

Will discuss this with @rohit-nayak-ps. Thank you for the bug report, @mvh-stripe !