Compatibility issues between v11 and v12 for getting workflows

[ejortegau]

Context

We run vitess v11 and are in the process to upgrade to v12. We are at a step in which vtctld hosts run v12 and tablets run v11.

Issue

Attempting to retrieve the list of workflows using vtctldClient.GetWorkflows() gives this error:

rpc error: code = Unknown desc = rpc error: code = Unknown desc = TabletManager.VReplicationExec on us_east_1a-0300036140 error: Unknown column 'tags' in 'field list' (errno 1054) (sqlstate 42S22) during query: select id, workflow, source, pos, stop_pos, max_replication_lag, state, db_name, time_updated, transaction_timestamp, message, tags from _vt.vreplication where db_name = 'vt_auth1': Unknown column 'tags' in 'field list' (errno 1054) (sqlstate 42S22) during query: select id, workflow, source, pos, stop_pos, max_replication_lag, state, db_name, time_updated, transaction_timestamp, message, tags from _vt.vreplication where db_name = 'vt_auth1'\nrpc error: code = Unknown desc = TabletManager.VReplicationExec on us_east_1b-0300092248 error: Unknown column 'tags' in 'field list' (errno 1054) (sqlstate 42S22) during query: select id, workflow, source, pos, stop_pos, max_replication_lag, state, db_name, time_updated, transaction_timestamp, message, tags from _vt.vreplication where db_name = 'vt_auth1': Unknown column 'tags' in 'field list' (errno 1054) (sqlstate 42S22) during query: select id, workflow, source, pos, stop_pos, max_replication_lag, state, db_name, time_updated, transaction_timestamp, message, tags from _vt.vreplication where db_name = 'vt_auth1'\nrpc error: code = Unknown desc = TabletManager.VReplicationExec on us_east_1d-0300142007 error: Unknown column 'tags' in 'field list' (errno 1054) (sqlstate 42S22) during query: select id, workflow, source, pos, stop_pos, max_replication_lag, state, db_name, time_updated, transaction_timestamp, message, tags from _vt.vreplication where db_name = 'vt_auth1': Unknown column 'tags' in 'field list' (errno 1054) (sqlstate 42S22) during query: select id, workflow, source, pos, stop_pos, max_replication_lag, state, db_name, time_updated, transaction_timestamp, message, tags from _vt.vreplication where db_name = 'vt_auth1'\nrpc error: code = Unknown desc = TabletManager.VReplicationExec on us_east_1e-0300205075 error: Unknown column 'tags' in 'field list' (errno 1054) (sqlstate 42S22) during query: select id, workflow, source, pos, stop_pos, max_replication_lag, state, db_name, time_updated, transaction_timestamp, message, tags from _vt.vreplication where db_name = 'vt_auth1': Unknown column 'tags' in 'field list' (errno 1054) (sqlstate 42S22) during query: select id, workflow, source, pos, stop_pos, max_replication_lag, state, db_name, time_updated, transaction_timestamp, message, tags from _vt.vreplication where db_name = 'vt_auth1'\nrpc error: code = Unknown desc = TabletManager.VReplicationExec on us_east_1e-0300210134 error: Unknown column 'tags' in 'field list' (errno 1054) (sqlstate 42S22) during query: select id, workflow, source, pos, stop_pos, max_replication_lag, state, db_name, time_updated, transaction_timestamp, message, tags from _vt.vreplication where db_name = 'vt_auth1': Unknown column 'tags' in 'field list' (errno 1054) (sqlstate 42S22) during query: select id, workflow, source, pos, stop_pos, max_replication_lag, state, db_name, time_updated, transaction_timestamp, message, tags from _vt.vreplication where db_name = 'vt_auth1'

Cause

This seems related to the caller expecting to query a tags column in _vt.vreplication which does not exist in vttablets running v11 as it was added in v12.

While docs do mention that it is recommended to upgrade first the tablets and then the vtctlds, informal conversations on slack indicate that, at least in principle, components should be able to interoperate with components one major version above and below. For our case, following the suggested upgrade order was not feasible from a fleet management point of view. Hence, it would be very good for us (and I suspect, any other large scale deployment) to be able to do it in other orders.

[frouioui]

Hey @ejortegau, is this issue still valid today with currently supported releases of Vitess?

[mattlord]

The general issue still remains. That's what this #14424 and the tracking issue it's part of are meant to address.