[Serve] [Doc] Add a tip for retry mechanism in scale-to-zero

[MaoZiming]

#3194 (comment)

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Any manual or new tests for this PR (please specify below)
• All smoke tests: pytest tests/test_smoke.py
• Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
• Backward compatibility tests: bash tests/backward_comaptibility_tests.sh

[cblmemo]

Could we explain why we need a retry mechanism?

[MaoZiming]

We mentioned "wait until the replicas are provisioned and ready"?

[Michaelvll]

Thanks for adding the tip in the doc @MaoZiming! Left a question of the autoscaler behavior. : )

[Michaelvll]

Suggested change

	If the scale-to-zero is set, the clients that access the endpoint should make sure to have a retry mechanism to be able to wait until the replicas are provisioned and ready.
	If the scale-to-zero is set, the clients that access the endpoint should make sure to have a retry mechanism to be able to wait until the replicas are provisioned and ready, i.e., starting a new replica when there is zero replica available.

Question: how does our autoscaler handle the case for retrying? If the client keeps retrying for failed requests, will those failed requests be considered for calculating the target_num_replicas? Need to think about the case, when zero replica is available and the client keeps retrying, causing the number of requests in the window become significantly large. Will it cause the autoscaler to scale up the service to max_replicas?

[MaoZiming]

Yes. The failed requests will contribute to calculating the target_num_replicas.
It might cause the autoscaler to scale up to max_replicas. As long as the retry frequency is not too big we should be fine

[Michaelvll]

Should we avoid the request to make target_num_replicas to be larger than 1 when there is no ready replicas?

[MaoZiming]

I still think maybe we should not - SkyServe as it is now cannot differentiate between retry requests and actual user requests. When there is no ready replica but accumulated requests, it is possible that there is a big increase in user requests and we might want to scale to more replicas (even though some of them might be retries) to be safe.