-
Notifications
You must be signed in to change notification settings - Fork 161
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature]: Support resources
with the gateway
configuration type
#1664
Comments
skypilot has this already. Customizing SkyServe controller resources Use a lower-cost controller. (if you have a few services running) Enforcing the controller to run on a specific location. This is particularly useful when you want the service endpoint within specific geographical region. (Default: cheapest location) Changing the maximum number of services that can be run concurrently, which is the minimum number between 4x the vCPUs of the controller and the memory in GiB of the controller. (Default: 16) Changing the disk_size of the controller to store more logs. (Default: 200GB) To achieve the above, you can specify custom configs in ~/.sky/config.yaml with the following fields: |
this issue is causing a bottle neck for users who have many services/users |
@Stealthwriter, dstack indeed uses too small gateway instances to handle such loads (e.g. "t2.micro" on AWS). If you run dstack from source, you can change the hardcoded instance type and disk_size here: dstack/src/dstack/_internal/core/backends/aws/compute.py Lines 275 to 277 in f654022
Please try changing them and provision a new gateway and see if the problem persist. If it helps, we're going to prioritize making instance_type for gateway configurable. |
@Stealthwriter, it is possible to create different gateways for different services to spread the load — just create and apply multiple gateway configurations. Choosing a gateway for a service is awkward now but it can be done by setting the default gateway before running the service. dstack gateway update --set-default gateway-1
dstack apply -f service-1.dstack.yml
dstack gateway update --set-default gateway-2
dstack apply -f service-2.dstack.yml
Note that different gateways should have different domain names. Running multiple replicas of the same gateway with the same domain name is not supported yet. |
good idea thanks |
resources
with the gateway
configuration type
Hello @Stealthwriter :) |
Hi @naaa760! Thank you for your interest in contributing to |
This issue is stale because it has been open for 30 days with no activity. |
If I send too many requests, like 2000 per second, the gateway can't handle that. Also, I have 9 services running. We need to either run multiple gateways or at least be able to increase instance resources.
Proposal:
Allow to specify
resources
for thegateway
configuration type –just like for runs (dev-environment
,task
, andservice
) configurations.The text was updated successfully, but these errors were encountered: