Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs: throttler, exempting apps #1549

Merged
merged 5 commits into from
Aug 2, 2023
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 17 additions & 1 deletion content/en/docs/18.0/reference/features/tablet-throttler.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ It is possible for the throttler to respond differently -- to some extent -- to

It is possible to _restrict_ the throttler's response to one or more apps. For example, it's possible to completely throttle "vreplication" while still responding `HTTP 200` to other apps. This is typically used to give way or precedence to one or two apps, or otherwise to further reduce the incoming load from a specific app.

It is _not possible_ to give an app more way than the throttler's standard behavior. That is, if the throttler is set to throttler at `5s` replication lag, it is _not possible_ to respond wih `HTTP 200` to a specific app with replication lag at `7s`.
Starting `v18`, it is also possible to _exempt_ an app from throttling, even if the throttler is otherwise rejecting requests with metrics beyond the threshold. This is a feature that the user will have to use with great care, and only in a situation where they absolutely have to give a specific workflow/migration the highest priority above all else. See discussion in examples, below.
shlomi-noach marked this conversation as resolved.
Show resolved Hide resolved

## Configuration

Expand Down Expand Up @@ -229,6 +229,22 @@ Fully throttle all Online DDL (schema changes) for the next hour and a half:
$ vtctldclient UpdateThrottlerConfig --throttle-app "online-ddl" --throttle-app-ratio=1.0 --throttle-app-duration "1h30m" commerce
```

Exempt `vreplication` from beign throttled, even if otherwise the metrics are past teh throttler threshold (e.g. replication lag is high):
shlomi-noach marked this conversation as resolved.
Show resolved Hide resolved

```sh
$ vtctldclient UpdateThrottlerConfig --throttle-app "vreplication" --throttle-app-duration "30m" --throttle-app-exempt commerce
```

Use the above with great care. Exempting one app can cause starvation to all other apps. Consider, for example, the common use case where throttling is based on replication lag. By exempting `vreplication`, it is free to grab all the resources it wants. It is possible and likely that it will drive replication lag higher than the threshold, which means all other throttler clients will be fully throttled and with all requests rejected.

Exemption times out just as other throttling rules. To unexempt, any of the following will do:
shlomi-noach marked this conversation as resolved.
Show resolved Hide resolved

```sh
$ vtctldclient UpdateThrottlerConfig --throttle-app "vreplication" --throttle-app-exempt=false commerce
$ vtctldclient UpdateThrottlerConfig --throttle-app "vreplication" --throttle-app-duration "0" commerce
$ vtctldclient UpdateThrottlerConfig --unthrottle-app "vreplication" commerce
```

### Information

Throttler configuration is pare of the `Keyspace` entry:
shlomi-noach marked this conversation as resolved.
Show resolved Hide resolved
Expand Down