Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Serve] [Doc] Add a tip for retry mechanism in scale-to-zero #3232

Merged
merged 3 commits into from
Feb 27, 2024
Merged

[Serve] [Doc] Add a tip for retry mechanism in scale-to-zero #3232

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6d0fc2d
autoscaling doc
MaoZiming Feb 26, 2024
a8144de
minor wording
MaoZiming Feb 26, 2024
e854a6e
Update docs/source/serving/autoscaling.rst
MaoZiming Feb 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/source/serving/autoscaling.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,3 +119,7 @@ If your service might experience long periods of time with no traffic, consider
# ...

The service will scale down all replicas when there is no traffic to the system and will save costs on idle replicas. When upscaling from zero, the upscale delay will be ignored in order to bring up the service faster.

.. tip::

If the scale-to-zero is set, the clients that access the endpoint should make sure to have a retry mechanism to be able to wait until the replicas are provisioned and ready, i.e., starting a new replica when there is zero replica available.
Loading