feat(ml): support dynamic scaling

[rkojedzinszky]

#12064

[mertalev]

I'd rather not have it make a TCP handshake for every request. There's surely a less crude way to make load balancing work for your environment.

[rkojedzinszky]

I understand your view here. Unfortunately, load-balancing happens only when establishing TCP connections. While the connections remain open, microservices server wont open new ones, thus no load-balancing will happen. Perhaps, we could make this behavior configurable, and not hardcode this into the image.

BTW, I could already make the container behave this way, by specifying GUNICORN_CMD_ARGS=--keep-alive 0 environment variable. But, it could be nicer if we had a dedicated environment variable for this behavior.

Right now, I am in the process of importing many photos. Thus, it happens that many ML requests are made after each other, so there is no way for kept-alive connections to be closed. In a load-balanced environment, I would really like to see that each query ends up on different server, in a round-robin or any fashion, that is up to the load-balancer. That is only achievable with connections being closed after a response.

[mertalev]

I'm fine with this being an environmental variable without changing the default behavior. Maybe MACHINE_LEARNING_KEEPALIVE_TIMEOUT_S?

[rkojedzinszky]

I have named the variable as MACHINE_LEARNING_HTTP_KEEPALIVE_TIMEOUT_S, to indicate it is a http-level timeout. This way it does not change default behavior, which is always a great thing to preserve.

[mertalev]

You should run npm i followed by prettier --fix . in the docs folder to fix the formatting.

[rkojedzinszky]

I will sure, thanks!