Merge #103757

103757: roach{prod,test}: add first-class support for disk snapshots r=irfansharif a=irfansharif

Part of #89978. Pre-cursor to #83826. Part of #98703.

Long-lived disk snapshots can drastically reduce testing time for scale tests. Tests, whether run by hand or through CI, need only run the long running fixture generating code (importing some dataset, generating it organically through workload, etc.) once snapshot fingerprints are changed, fingerprints that incorporate the major crdb version that generated them.

Here's an example run that freshly generates disk snapshots:

    === RUN   admission-control/index-backfill
    no existing snapshots found for admission-control/index-backfill (ac-index-backfill), doing pre-work
    created volume snapshot ac-index-backfill-0001-vunknown-1-n2-standard-8 for volume irfansharif-snapshot-0001-1 on irfansharif-snapshot-0001-1/n1
    using 1 newly created snapshot(s) with prefix "ac-index-backfill"
    detached and deleted volume irfansharif-snapshot-0001-1 from irfansharif-snapshot-0001
    created volume irfansharif-snapshot-0001-1
    attached volume irfansharif-snapshot-0001-1 to irfansharif-snapshot-0001
    mounted irfansharif-snapshot-0001-1 to irfansharif-snapshot-0001
    --- PASS: admission-control/index-backfill (79.14s)

Here's a subsequent run that makes use of the aforementioned disk snapshots:

    === RUN   admission-control/index-backfill
    using 1 pre-existing snapshot(s) with prefix "ac-index-backfill"
    detached and deleted volume irfansharif-snapshot-0001-1 from irfansharif-snapshot-0001
    created volume irfansharif-snapshot-0001-1
    attached volume irfansharif-snapshot-0001-1 to irfansharif-snapshot-0001
    mounted irfansharif-snapshot-0001-1 to irfansharif-snapshot-0001
    --- PASS: admission-control/index-backfill (43.47s)

We add the following APIs to the roachtest.Cluster interface, for tests to interact with disk snapshots. admission-control/index-backfill is an example test making use of these APIs.
```go
  type Cluster interface {
      // ...

      // CreateSnapshot creates volume snapshots of the cluster using
      // the given prefix. These snapshots can later be retrieved,
      // deleted or applied to already instantiated clusters.
      CreateSnapshot(ctx context.Context, snapshotPrefix string) error

      // ListSnapshots lists the individual volume snapshots that
      // satisfy the search criteria.
      ListSnapshots(
        ctx context.Context, vslo vm.VolumeSnapshotListOpts,
      ) ([]vm.VolumeSnapshot, error)

      // DeleteSnapshots permanently deletes the given snapshots.
      DeleteSnapshots(
        ctx context.Context, snapshots ...vm.VolumeSnapshot,
      ) error

      // ApplySnapshots applies the given volume snapshots to the
      // underlying cluster. This is a destructive operation as far as
      // existing state is concerned - all already-attached volumes are
      // detached and deleted to make room for new snapshot-derived
      // volumes. The new volumes are created using the same specs
      // (size, disk type, etc.) as the original cluster.
      ApplySnapshots(
        ctx context.Context, snapshots []vm.VolumeSnapshot,
      ) error
  }
```
These Cluster APIs are in turn is powered by the following additions to the vm.Provider interface, implemented by each cloud provider. GCE is the fully spec-ed out one for now.
```go
  type Provider interface {
      // ...

      // CreateVolume creates a new volume using the given options.
      CreateVolume(l *logger.Logger, vco VolumeCreateOpts) (Volume, error)

      // ListVolumes lists all volumes already attached to the given VM.
      ListVolumes(l *logger.Logger, vm *VM) ([]Volume, error)

      // DeleteVolume detaches and deletes the given volume from the
      // given VM.
      DeleteVolume(l *logger.Logger, volume Volume, vm *VM) error

      // AttachVolume attaches the given volume to the given VM.
      AttachVolume(l *logger.Logger, volume Volume, vm *VM) (string, error)

      // CreateVolumeSnapshot creates a snapshot of the given volume,
      // using the given options.
      CreateVolumeSnapshot(
        l *logger.Logger, volume Volume, vsco VolumeSnapshotCreateOpts,
      ) (VolumeSnapshot, error)

      // ListVolumeSnapshots lists the individual volume snapshots that
      // satisfy the search criteria.
      ListVolumeSnapshots(
        l *logger.Logger, vslo VolumeSnapshotListOpts,
      ) ([]VolumeSnapshot, error)

      // DeleteVolumeSnapshot permanently deletes the given snapshot.
      DeleteVolumeSnapshot(l *logger.Logger, snapshot VolumeSnapshot) error
  }
```
Since these snapshots necessarily outlive the tests, and we don't want them dangling perpetually, we introduce a prune-dangling roachtest that acts as a poor man's cron job, sifting through expired snapshots (>30days) and deleting them. For GCE at least it's not obvious to me how to create these snapshots in cloud buckets with a TTL built in, hence this hack. It looks like this (with change to the TTL):

    === RUN   prune-dangling
    pruned old snapshot ac-index-backfill-0001-vunknown-1-n2-standard-8
    --- PASS: prune-dangling (8.59s)

---

We add expose some of these APIs through the roachprod binary directly.
```
$ roachprod snapshot --help
  snapshot enables creating/listing/deleting/applying cluster snapshots

  Usage:
    roachprod snapshot [command]

  Available Commands:
    create      snapshot a named cluster, using the given snapshot name and description
    list        list all snapshots for the given cloud provider, optionally filtering by the given name
    delete      delete all snapshots for the given cloud provider optionally filtering by the given name
    apply       apply the named snapshots from the given cloud provider to the named cluster
```
---

About admission-control/index-backfill. It's a fully featured test that uses the TPC-C 100k dataset and runs a foreground load for 20k customers. It takes >4hrs to import this data set; with disk snapshots this step is skipped entirely and takes a few minutes. The actual test is trivial, we run the foreground load for 1hr and run a large index backfill concurrently. Before #98308, this results in wild performance oscillations. It's still a bit wild after flow control, but less so.

We slightly extend the tpc-e harness to make this happen, adding a few smarts: exposing a 'during' helper to run backfills concurrently with foreground load, integrate with --skip-init, estimated setup times, prometheus, and disk snapshots of course.

Release note: None

Co-authored-by: irfan sharif <[email protected]>

pkg/cmd/roachprod/flags.go

@@ -267,6 +267,13 @@ Default is "RECURRING '*/15 * * * *' FULL BACKUP '@hourly' WITH SCHEDULE OPTIONS
 	initCmd.Flags().IntVar(&startOpts.InitTarget,
 		"init-target", startOpts.InitTarget, "node on which to run initialization")
 
+	snapshotDeleteCmd.Flags().BoolVar(&dryrun,
+		"dry-run", false, "dry run (don't perform any actions)")
+	snapshotCmd.AddCommand(snapshotCreateCmd)
+	snapshotCmd.AddCommand(snapshotListCmd)
+	snapshotCmd.AddCommand(snapshotDeleteCmd)
+	snapshotCmd.AddCommand(snapshotApplyCmd)
+
 	rootStorageCmd.AddCommand(rootStorageCollectionCmd)
 	rootStorageCollectionCmd.AddCommand(collectionStartCmd)
 	rootStorageCollectionCmd.AddCommand(collectionStopCmd)

pkg/cmd/roachprod/main.go

@@ -580,6 +580,7 @@ The "status" command outputs the binary and PID for the specified nodes:
 			if status.Err != nil {
 				config.Logger.Printf("  %2d: %s %s\n", status.NodeID, status.Err.Error())
 			} else if !status.Running {
+				// TODO(irfansharif): Surface the staged version here?
 				config.Logger.Printf("  %2d: not running\n", status.NodeID)
 			} else {
 				config.Logger.Printf("  %2d: %s %s\n", status.NodeID, status.Version, status.Pid)
@@ -1056,6 +1057,113 @@ var grafanaURLCmd = &cobra.Command{
 	}),
 }
 
+var snapshotCmd = &cobra.Command{
+	Use:   `snapshot`,
+	Short: "snapshot enables creating/listing/deleting/applying cluster snapshots",
+	Args:  cobra.MinimumNArgs(1),
+}
+
+var snapshotCreateCmd = &cobra.Command{
+	Use:   `create <cluster> <name> <description>`,
+	Short: "snapshot a named cluster, using the given snapshot name and description",
+	Args:  cobra.ExactArgs(3),
+	Run: wrap(func(cmd *cobra.Command, args []string) error {
+		cluster := args[0]
+		name := args[1]
+		desc := args[2]
+		snapshots, err := roachprod.CreateSnapshot(context.Background(), config.Logger, cluster, vm.VolumeSnapshotCreateOpts{
+			Name:        name,
+			Description: desc,
+		})
+		if err != nil {
+			return err
+		}
+		for _, snapshot := range snapshots {
+			config.Logger.Printf("created snapshot %s (id: %s)", snapshot.Name, snapshot.ID)
+		}
+		return nil
+	}),
+}
+
+var snapshotListCmd = &cobra.Command{
+	Use:   `list <provider> [<name>]`,
+	Short: "list all snapshots for the given cloud provider, optionally filtering by the given name",
+	Args:  cobra.RangeArgs(1, 2),
+	Run: wrap(func(cmd *cobra.Command, args []string) error {
+		provider := args[0]
+		var name string
+		if len(args) == 2 {
+			name = args[1]
+		}
+		snapshots, err := roachprod.ListSnapshots(context.Background(), config.Logger, provider,
+			vm.VolumeSnapshotListOpts{
+				NamePrefix: name,
+			},
+		)
+		if err != nil {
+			return err
+		}
+		for _, snapshot := range snapshots {
+			config.Logger.Printf("found snapshot %s (id: %s)", snapshot.Name, snapshot.ID)
+		}
+		return nil
+	}),
+}
+
+var snapshotDeleteCmd = &cobra.Command{
+	Use:   `delete <provider> <name>`,
+	Short: "delete all snapshots for the given cloud provider optionally filtering by the given name",
+	Args:  cobra.ExactArgs(2),
+	Run: wrap(func(cmd *cobra.Command, args []string) error {
+		ctx := context.Background()
+		provider, name := args[0], args[1]
+		snapshots, err := roachprod.ListSnapshots(ctx, config.Logger, provider,
+			vm.VolumeSnapshotListOpts{
+				NamePrefix: name,
+			},
+		)
+		if err != nil {
+			return err
+		}
+
+		for _, snapshot := range snapshots {
+			config.Logger.Printf("deleting snapshot %s (id: %s)", snapshot.Name, snapshot.ID)
+		}
+		if !dryrun {
+			if err := roachprod.DeleteSnapshots(ctx, config.Logger, provider, snapshots...); err != nil {
+				return err
+			}
+		}
+		config.Logger.Printf("done")
+		return nil
+	}),
+}
+
+var snapshotApplyCmd = &cobra.Command{
+	Use:   `apply <provider> <name> <cluster> `,
+	Short: "apply the named snapshots from the given cloud provider to the named cluster",
+	Args:  cobra.ExactArgs(3),
+	Run: wrap(func(cmd *cobra.Command, args []string) error {
+		ctx := context.Background()
+		provider, name, cluster := args[0], args[1], args[2]
+		snapshots, err := roachprod.ListSnapshots(ctx, config.Logger, provider,
+			vm.VolumeSnapshotListOpts{
+				NamePrefix: name,
+			},
+		)
+		if err != nil {
+			return err
+		}
+
+		return roachprod.ApplySnapshots(ctx, config.Logger, cluster, snapshots, vm.VolumeCreateOpts{
+			Size: 500, // TODO(irfansharif): Make this configurable?
+			Labels: map[string]string{
+				vm.TagUsage: "roachprod",
+			},
+		})
+	}),
+}
+
 var rootStorageCmd = &cobra.Command{
 	Use:   `storage`,
 	Short: "storage enables administering storage related commands and configurations",
@@ -1126,7 +1234,11 @@ var storageSnapshotCmd = &cobra.Command{
 		cluster := args[0]
 		name := args[1]
 		desc := args[2]
-		return roachprod.SnapshotVolume(context.Background(), config.Logger, cluster, name, desc)
+		_, err := roachprod.CreateSnapshot(context.Background(), config.Logger, cluster, vm.VolumeSnapshotCreateOpts{
+			Name:        name,
+			Description: desc,
+		})
+		return err
 	}),
 }
 
@@ -1219,6 +1331,7 @@ func main() {
 		grafanaDumpCmd,
 		grafanaURLCmd,
 		rootStorageCmd,
+		snapshotCmd,
 		fixLongRunningAWSHostnamesCmd,
 	)
 	setBashCompletionFunction()

pkg/cmd/roachtest/cluster.go

@@ -573,6 +573,9 @@ func MachineTypeToCPUs(s string) int {
 		if _, err := fmt.Sscanf(s, "n1-standard-%d", &v); err == nil {
 			return v
 		}
+		if _, err := fmt.Sscanf(s, "n2-standard-%d", &v); err == nil {
+			return v
+		}
 		if _, err := fmt.Sscanf(s, "n1-highcpu-%d", &v); err == nil {
 			return v
 		}
@@ -1689,6 +1692,39 @@ func (c *clusterImpl) doDestroy(ctx context.Context, l *logger.Logger) <-chan st
 	return ch
 }
 
+func (c *clusterImpl) ListSnapshots(
+	ctx context.Context, vslo vm.VolumeSnapshotListOpts,
+) ([]vm.VolumeSnapshot, error) {
+	return roachprod.ListSnapshots(ctx, c.l, c.spec.Cloud, vslo)
+}
+
+func (c *clusterImpl) DeleteSnapshots(ctx context.Context, snapshots ...vm.VolumeSnapshot) error {
+	return roachprod.DeleteSnapshots(ctx, c.l, c.spec.Cloud, snapshots...)
+}
+
+func (c *clusterImpl) CreateSnapshot(
+	ctx context.Context, snapshotPrefix string,
+) ([]vm.VolumeSnapshot, error) {
+	return roachprod.CreateSnapshot(ctx, c.l, c.name, vm.VolumeSnapshotCreateOpts{
+		Name:        snapshotPrefix,
+		Description: fmt.Sprintf("snapshot for test: %s", c.t.Name()),
+		Labels: map[string]string{
+			vm.TagUsage: "roachtest",
+		},
+	})
+}
+
+func (c *clusterImpl) ApplySnapshots(ctx context.Context, snapshots []vm.VolumeSnapshot) error {
+	opts := vm.VolumeCreateOpts{
+		Size: c.spec.VolumeSize,
+		Type: c.spec.GCEVolumeType, // TODO(irfansharif): This is only applicable to GCE. Change that.
+		Labels: map[string]string{
+			vm.TagUsage: "roachtest",
+		},
+	}
+	return roachprod.ApplySnapshots(ctx, c.l, c.name, snapshots, opts)
+}
+
 // Put a local file to all of the machines in a cluster.
 // Put is DEPRECATED. Use PutE instead.
 func (c *clusterImpl) Put(ctx context.Context, src, dest string, nodes ...option.Option) {

pkg/cmd/roachtest/cluster/BUILD.bazel

@@ -16,6 +16,7 @@ go_library(
         "//pkg/roachprod/install",
         "//pkg/roachprod/logger",
         "//pkg/roachprod/prometheus",
+        "//pkg/roachprod/vm",
         "@com_github_cockroachdb_errors//:errors",
     ],
 )

pkg/cmd/roachtest/cluster/cluster_interface.go

@@ -20,6 +20,7 @@ import (
 	"github.com/cockroachdb/cockroach/pkg/roachprod/install"
 	"github.com/cockroachdb/cockroach/pkg/roachprod/logger"
 	"github.com/cockroachdb/cockroach/pkg/roachprod/prometheus"
+	"github.com/cockroachdb/cockroach/pkg/roachprod/vm"
 )
 
 // Cluster is the interface through which a given roachtest interacts with the
@@ -134,4 +135,34 @@ type Cluster interface {
 
 	StartGrafana(ctx context.Context, l *logger.Logger, promCfg *prometheus.Config) error
 	StopGrafana(ctx context.Context, l *logger.Logger, dumpDir string) error
+
+	// Volume snapshot related APIs.
+	//
+	// NB: The --local case for these snapshot APIs are that they all no-op. But
+	// it should be transparent to roachtests. The assumption this interface
+	// makes is that the calling roachtest that needed to CreateSnapshot will
+	// proceed with then using the already populated disks. Disks that it
+	// populated having not found any existing snapshots. So --local runs don't
+	// rely on the remaining snapshot methods to actually do anything.
+
+	// CreateSnapshot creates volume snapshots of the cluster using the given
+	// prefix. These snapshots can later be retrieved, deleted or applied to
+	// already instantiated clusters.
+	//
+	CreateSnapshot(ctx context.Context, snapshotPrefix string) ([]vm.VolumeSnapshot, error)
+	// ListSnapshots lists the individual volume snapshots that satisfy the
+	// search criteria.
+	ListSnapshots(ctx context.Context, vslo vm.VolumeSnapshotListOpts) ([]vm.VolumeSnapshot, error)
+	// DeleteSnapshots permanently deletes the given snapshots.
+	DeleteSnapshots(ctx context.Context, snapshots ...vm.VolumeSnapshot) error
+	// ApplySnapshots applies the given volume snapshots to the underlying
+	// cluster. This is a destructive operation as far as existing state is
+	// concerned - all already-attached volumes are detached and deleted to make
+	// room for new snapshot-derived volumes. The new volumes are created using
+	// the same specs (size, disk type, etc.) as the original cluster.
+	//
+	// TODO(irfansharif): The implementation tacitly assumes one volume
+	// per-node, but this could be changed. Another assumption is that all
+	// volumes are created identically.
+	ApplySnapshots(ctx context.Context, snapshots []vm.VolumeSnapshot) error
 }

pkg/cmd/roachtest/cluster_test.go

@@ -58,6 +58,10 @@ type testWrapper struct {
 	l *logger.Logger
 }
 
+func (t testWrapper) SnapshotPrefix() string {
+	return ""
+}
+
 func (t testWrapper) BuildVersion() *version.Version {
 	panic("implement me")
 }

pkg/cmd/roachtest/registry/test_spec.go

@@ -102,6 +102,11 @@ type TestSpec struct {
 	// True iff results from this test should not be published externally,
 	// e.g. to GitHub.
 	RedactResults bool
+
+	// SnapshotPrefix is set by tests that make use of volume snapshots.
+	// Prefixes must be not only be unique across tests, but one cannot be a
+	// prefix of another.
+	SnapshotPrefix string
 }
 
 // PostValidation is a type of post-validation that runs after a test completes.

pkg/cmd/roachtest/spec/cluster_spec.go

@@ -85,6 +85,13 @@ type ClusterSpec struct {
 	RandomlyUseZfs bool
 
 	GatherCores bool
+
+	// GCE-specific arguments.
+	//
+	// TODO(irfansharif): This cluster spec type suffers the curse of
+	// generality. Make it easier to just inject cloud-specific arguments.
+	GCEMinCPUPlatform string
+	GCEVolumeType     string
 }
 
 // MakeClusterSpec makes a ClusterSpec.
@@ -155,6 +162,7 @@ func getGCEOpts(
 	localSSD bool,
 	RAID0 bool,
 	terminateOnMigration bool,
+	minCPUPlatform, volumeType string,
 ) vm.ProviderOpts {
 	opts := gce.DefaultProviderOpts()
 	opts.MachineType = machineType
@@ -173,6 +181,8 @@ func getGCEOpts(
 		opts.UseMultipleDisks = !RAID0
 	}
 	opts.TerminateOnMigration = terminateOnMigration
+	opts.MinCPUPlatform = minCPUPlatform
+	opts.PDVolumeType = volumeType
 
 	return opts
 }
@@ -289,7 +299,9 @@ func (s *ClusterSpec) RoachprodOpts(
 		providerOpts = getAWSOpts(machineType, zones, s.VolumeSize, createVMOpts.SSDOpts.UseLocalSSD)
 	case GCE:
 		providerOpts = getGCEOpts(machineType, zones, s.VolumeSize, ssdCount,
-			createVMOpts.SSDOpts.UseLocalSSD, s.RAID0, s.TerminateOnMigration)
+			createVMOpts.SSDOpts.UseLocalSSD, s.RAID0, s.TerminateOnMigration,
+			s.GCEMinCPUPlatform, s.GCEVolumeType,
+		)
 	case Azure:
 		providerOpts = getAzureOpts(machineType, zones)
 	}

pkg/cmd/roachtest/spec/option.go

@@ -17,6 +17,17 @@ type Option interface {
 	apply(spec *ClusterSpec)
 }
 
+type cloudOption string
+
+func (o cloudOption) apply(spec *ClusterSpec) {
+	spec.Cloud = string(o)
+}
+
+// Cloud controls what cloud is used to create the cluster.
+func Cloud(s string) Option {
+	return cloudOption(s)
+}
+
 type nodeCPUOption int
 
 func (o nodeCPUOption) apply(spec *ClusterSpec) {

pkg/cmd/roachtest/test/test_interface.go

@@ -25,8 +25,9 @@ type Test interface {
 	Name() string
 	BuildVersion() *version.Version
 	IsBuildVersion(string) bool // "vXX.YY"
+	SnapshotPrefix() string
 	Helper()
-	// Spec() returns the *registry.TestSpec as an interface{}.
+	// Spec returns the *registry.TestSpec as an interface{}.
 	//
 	// TODO(tbg): cleaning this up is mildly tricky. TestSpec has the Run field
 	// which depends both on `test` (and `cluster`, though this matters less), so

pkg/cmd/roachtest/test_impl.go

@@ -154,6 +154,10 @@ func (t *testImpl) Name() string {
 	return t.spec.Name
 }
 
+func (t *testImpl) SnapshotPrefix() string {
+	return t.spec.SnapshotPrefix
+}
+
 // L returns the test's logger.
 func (t *testImpl) L() *logger.Logger {
 	return t.l