Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature]: Support resources with the gateway configuration type #1664

Open
Tracked by #1735
Stealthwriter opened this issue Sep 5, 2024 · 8 comments
Open
Tracked by #1735
Labels

Comments

@Stealthwriter
Copy link

Stealthwriter commented Sep 5, 2024

If I send too many requests, like 2000 per second, the gateway can't handle that. Also, I have 9 services running. We need to either run multiple gateways or at least be able to increase instance resources.

Proposal:

Allow to specify resources for the gateway configuration type –just like for runs (dev-environment, task, and service) configurations.

@Stealthwriter Stealthwriter added the bug Something isn't working label Sep 5, 2024
@Stealthwriter
Copy link
Author

skypilot has this already.

Customizing SkyServe controller resources
You may want to customize the resources of the SkyServe controller for several reasons:

Use a lower-cost controller. (if you have a few services running)

Enforcing the controller to run on a specific location. This is particularly useful when you want the service endpoint within specific geographical region. (Default: cheapest location)

Changing the maximum number of services that can be run concurrently, which is the minimum number between 4x the vCPUs of the controller and the memory in GiB of the controller. (Default: 16)

Changing the disk_size of the controller to store more logs. (Default: 200GB)

To achieve the above, you can specify custom configs in ~/.sky/config.yaml with the following fields:

@Stealthwriter
Copy link
Author

this issue is causing a bottle neck for users who have many services/users

@r4victor
Copy link
Collaborator

r4victor commented Sep 6, 2024

@Stealthwriter, dstack indeed uses too small gateway instances to handle such loads (e.g. "t2.micro" on AWS).

If you run dstack from source, you can change the hardcoded instance type and disk_size here:

disk_size=10,
image_id=aws_resources.get_gateway_image_id(ec2_client),
instance_type="t2.micro",

Please try changing them and provision a new gateway and see if the problem persist. If it helps, we're going to prioritize making instance_type for gateway configurable.

@jvstme
Copy link
Collaborator

jvstme commented Sep 6, 2024

We need to be able to run multiple gateways

@Stealthwriter, it is possible to create different gateways for different services to spread the load — just create and apply multiple gateway configurations. Choosing a gateway for a service is awkward now but it can be done by setting the default gateway before running the service.

dstack gateway update --set-default gateway-1
dstack apply -f service-1.dstack.yml
dstack gateway update --set-default gateway-2
dstack apply -f service-2.dstack.yml

dstack will likely introduce a better interface for choosing between gateways as part of #1595.

Note that different gateways should have different domain names. Running multiple replicas of the same gateway with the same domain name is not supported yet.

@Stealthwriter
Copy link
Author

good idea thanks

@peterschmidt85 peterschmidt85 changed the title [Bug]: We should be able to customize Gateway instance, like choosing a bigger instance [Bug]: Allow to configure gateway instance type Sep 12, 2024
@peterschmidt85 peterschmidt85 mentioned this issue Sep 12, 2024
42 tasks
@peterschmidt85 peterschmidt85 added hacktoberfest and removed bug Something isn't working labels Sep 27, 2024
@peterschmidt85 peterschmidt85 changed the title [Bug]: Allow to configure gateway instance type [Feature]: Support resources with the gateway configuration type Sep 28, 2024
@naaa760
Copy link

naaa760 commented Oct 25, 2024

Hello @Stealthwriter :)
I want to contribute to this issue. Please assign this issue to me.
Thank you!

@jvstme
Copy link
Collaborator

jvstme commented Oct 28, 2024

Hi @naaa760! Thank you for your interest in contributing to dstack.
Assignment is not necessary, so please feel free to submit a PR if you know how to implement this. Note that this is a major feature that will require changes and testing in all dstack backends that support gateways: AWS, Azure, GCP, and Kubernetes.

Copy link

github-actions bot commented Dec 6, 2024

This issue is stale because it has been open for 30 days with no activity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

5 participants