executor: kill tidb [session id] can't stop executors and release resources quickly

[qw4990]

What problem does this PR solve?

fix #9831

What is changed and how it works?

Introduce execCtxWatcher to watch the context specified in Open and stop the wrapped executor when this context is cancelled.

Check List

Tests

• Unit test

[codecov]

Codecov Report

Merging #9844 into master will increase coverage by 0.9053%.
The diff coverage is 25%.

@@               Coverage Diff                @@
##             master      #9844        +/-   ##
================================================
+ Coverage   77.2161%   78.1214%   +0.9053%     
================================================
  Files           405        405                
  Lines         81641      85330      +3689     
================================================
+ Hits          63040      66661      +3621     
+ Misses        13931      13861        -70     
- Partials       4670       4808       +138

[lysu]

@qw4990 maybe we can pull this watcher goroutine "up" to connection level~?

for current code, it will create a new goroutine for each stmt but maybe better just create one watcher goroutine per connection?

tidb/server/conn.go

Line 706 in a2d4bd1

ctx1, cancelFunc := context.WithCancel(ctx)

seems be little stranger, does we just cancel a "connection ctx" and then close "current Executor" is enough? and no need to withCancel for each request?

cc @tiancaiamao

[tiancaiamao]

Why we need a additional watcher to do that... each executor should handle its ctx.Done, if not, there must be something wrong.

[qw4990]

@tiancaiamao Each executor can also handle this ctx.Done too, which is not conflicted with this global watcher as long as it's Close is reentrant.
But using a global watcher may be clearer and safer, since it's tedious to make all executors handle the ctx.Done and there is always a risk for forgetting to do it when adding new executors.

[tiancaiamao]

as long as it's Close is reentrant

I doubt this assumption, could we close the executor multiple times?

But using a global watcher may be clearer and safer

And poor performance if we add a goroutine for each statement

[qw4990]

as long as it's Close is reentrant

I doubt this assumption, could we close the executor multiple times?

But using a global watcher may be clearer and safer

And poor performance if we add a goroutine for each statement

But if all executors watch the context by themselves, there will be more extra goroutines than this one.

So how about we don't let other executors watch it and just use this global goroutine to control them all.

[tiancaiamao]

The watcher goroutine is not the key point.
We can let the kill tidb [session id] connection goroutine to serve as the watcher and call the executor.Close method (to kill another session).

The fundamental problem is that CAN WE CALL executor.Close ANY TIME WHILE THE EXECUTOR IS RUNNING...

In the old protocol, executor handle the ctx.Done() is optional.
It means that when a executor doesn't obey the rule, the worst case is just as the title says: "kill tidb [session id] can't stop executors and release resources quickly".
After the change, I'm not sure will executor be close twice, will there be data race when two kill request arrive at the same time.

If executor.Close is reentrant, I think your way works. @qw4990
what's your opinion @zz-jason

[zz-jason]

I'm afraid that executor.Close() is not thread-safe. Data race may happen when it is called by two session concurrently.

Close() an executor multiple times in a single thread is safe I think. Besides, the way @qw4990 used can prevent the executor being closed more than once.

[zz-jason]

CAN WE CALL executor.Close ANY TIME WHILE THE EXECUTOR IS RUNNING

Sure, we can

[qw4990]

We can let the kill tidb [session id] connection goroutine to serve as the watcher and call the executor.Close method (to kill another session).

It's hard to get other sessions' resources like executors in the kill tidb [session id] connection goroutine.
And it may be danger to manipulate a session's resource directly in another session's goroutine.
Since we have a context, just cancelling it and letting its session release resources by itself is more safe and go-style.
So, a watcher goroutine is essential.

If executor.Close is reentrant

All calls of Close are from top to bottom and the execCtxWatcher is wrapped around the root executor, so it will prevent executors from closing multiple times.

[tiancaiamao]

And it may be danger to manipulate a session's resource directly in another session's goroutine.

The watcher is doing the same thing, right?

So, a watcher goroutine is essential.

No, it's easy to set the cancel function (which is changed to executor.Close now) here:

https://github.com/pingcap/tidb/blob/f52fa9d9659b0fbeb0a0cb001c3046dbd0f5af18/executor/adapter.go#L241

And when one session receive the kill statement, it executes the cancelFunc ( actually executor.Close) to close current session's execution.

@qw4990

[qw4990]

According to your comments, the way to solve this problem has been changed.
Now it's clearer and lighter and no the extra goroutine.
PTAL~
@lysu @zz-jason @tiancaiamao @XuHuaiyu

[tiancaiamao]

Rest LGTM
PTAL @zz-jason

[zz-jason]

LGTM

[zz-jason]

/run-all-tests

[zz-jason]

@qw4990 please cherry pick this PR to release 2.1