A transaction may be committed and TiDB doesn't return an undetermined error

[youjiali1995]

Bug Report

A region error doesn't mean a write request is failed:

• When TiKV is gracefully shutdown, it may return StaleCommand to all proposed requests(gRPC server is shutdown asynchronously): https://github.com/tikv/tikv/blob/4ed382c21699596a84aa3342b27e3221c0741893/components/raftstore/src/store/fsm/apply.rs#L3006-L3010
• When a region is destroyed, it will return RegionNotFound to all proposed requests: https://github.com/tikv/tikv/blob/4ed382c21699596a84aa3342b27e3221c0741893/components/raftstore/src/store/fsm/apply.rs#L1132-L1144

Now TiDB sets undetermined errors only when encounters RPC errors. It's possible a transaction is committed and TiDB doesn't return an undetermined error. Fortunately, TiDB cleans up the primary lock first, so it doesn't break the atomicity of transaction.

[youjiali1995]

/cc @cfzjywxk @MyonKeminta @sticnarf @nrc

[youjiali1995]

When TiKV is gracefully shutdown, it may return StaleCommand to all proposed requests(gRPC server is shutdown asynchronously): https://github.com/tikv/tikv/blob/4ed382c21699596a84aa3342b27e3221c0741893/components/raftstore/src/store/fsm/apply.rs#L3006-L3010

We can shut down the gRPC server synchronously.

When a region is destroyed, it will return RegionNotFound to all proposed requests: https://github.com/tikv/tikv/blob/4ed382c21699596a84aa3342b27e3221c0741893/components/raftstore/src/store/fsm/apply.rs#L1132-L1144

I'm not clear under what circumstances this case will occur now. I think we should list all cases which may result in undetermined stats.

[sticnarf]

The second case looks unlikely to happen. Transfer and region merge should not result in undetermined states before destroying the peer.
If a peer destroys itself after being orphan like tikv/tikv#9113, the problem can happen. But it shouldn't be a problem if we add timeout to callback.