Feature Request: emit tablet type metric from vttablet to Prometheus

[maxenglander]

Feature Description

Emit a metric, available in Prometheus, like vttablet_tablet_type metric to indicate if that tablet target type is currently primary, replica, etc. Alternately include tablet type in existing status metrics like vttablet_tablet_state.

Use Case(s)

Use case is to detect when the primary is not in serving state. We have vttablet_tablet_state metric for that, but it does not include the tablet type.

The vttablet_tablet_type_count metric doesn't tell us what the current target type of the tablet is, so it doesn't help very much with this use case. The TabletType stat does have what we want, but it is not exported to Prometheus.

Metrics recently added by @rafer can very nearly be used to satisfy this use case: #13521. However, those metrics are not updated when there are no queries, nor in the case of FailedPrecondition errors like wrong tablet type:

vitess/go/vt/vttablet/tabletserver/state_manager.go

Lines 386 to 400 in a5e819d

	if sm.state != StateServing || !sm.replHealthy {
	// This specific error string needs to be returned for vtgate buffering to work.
	return vterrors.New(vtrpcpb.Code_CLUSTER_EVENT, vterrors.NotServing)
	}
	shuttingDown := sm.wantState != StateServing
	if shuttingDown && !allowOnShutdown {
	// This specific error string needs to be returned for vtgate buffering to work.
	return vterrors.New(vtrpcpb.Code_CLUSTER_EVENT, vterrors.ShuttingDown)
	}
	err = sm.verifyTargetLocked(ctx, target)
	if err != nil {
	return err
	}