[Roadmap] Q3 2024

[peterschmidt85]

This issue outlines the major items planned for Q3 2024.

For major bugs, see major

Core features

• dstack apply
  • dstack apply should be in the attached mode by default for any configuration type #1571
  • [UX] Show a plan for applying a fleet #1472
  • [UX] There is no --detach anymore with dstack apply #1671
  • Implement dstack apply for gateways #1223
  • Support running run configurations with dstack apply #1326
  • Introduce Fleets for cluster provisioning and management #1327
• Production-grade deployment of the server
  • [Feature]: CloudWatch Logs storage #1577
  • Support multi-replica server #1632
  • Support running dstack server with Postgres #1393
• Gateways
  • [Feature]: Support resources with the gateway configuration type #1664
  • [Feature]: Allow running services and models without a gateway #1595
  • [Feature]: Continuous health checks on services with reactions like pod-replacements  #1349
  • [Feature]: Initial support for routers (using AWS Bedrock) #1631
• Volumes
  • [Feature]: Support dstack volumes #1158
  • [Feature]: Support Runpod network volumes #1296
  • Support volumes for GCP #1469
  • Support multi-attach volumes #1467
• Private subnets
  • Support provisioning instances without public IPs on AWS #1201
  • Support internal-facing gateways on AWS #1171
  • Support gateways behind ALB with ACM certificate #1264
  • Support specifying non-default VPC for GCP backend #1304
  • Support provisioning instances without public IPs on GCP #1321
  • [Feature] Support private subnets on Azure #1672
• Clusters
  • Support multi-host TPUs #1337
  • Introduce Fleets for cluster provisioning and management #1327
  • Batch cluster provisioning #1489
• Implement secrets #708
• Reserved instances
  • [Feature]: Support the Capacity Blocks for ML feature for AWS #1155
• Add the control plane UI #1419

Supported architectures

• TPU
  • [Feature]: Add TPU support #956
  • Support multi-host TPUs #1337
• AMD
  • Support AMD with SSH fleets #1413
• Gaudi

Examples

Important

Community help is welcome!

• AMD
  • Publish the blog post and examples about AMD on RunPod #1598
  • Axolotl
  • TRL
• Nim
• GitHub Actions
• VLLM/TGI
• Llama 3.2 (multi-modal)
• FLUX
• Ray
• Spark
• Unsloth
• Alignment Handbook with Llama 3.1
• Nemo
• TensorRT-LLM
• Triton
• Function calling
• Llama Index
• LangChain
• TPU
• Multi-node Alignment Handbook
• Llama 3.1
• Fine-tuning Llama 3.1
• Axolotl

Improvements

• Support for H100 in AWS/GCP/Azure [Feature]: Support H100 with AWS #1238 [Feature]: Support H100 with GCP #1240 [Feature]: Support H100 with Azure #1239
• Support for H200 in AWS/GCP/Azure once they support it
• Support for L4 in AWS Support NVIDIA L4 with AWS (g6.* instance type) #1235
• Troubleshooting guide dstack server must collect logs from gateways and instances #1673
• Make it easier to add custom backends (implementation and documentation)

Research

Important

Research and feedback is required!

• Metrics: Research whether dstack should collect certain metrics out of the box (at least hardware utilization) or if it should be integrated with more enterprise-grade tools.
• Fault-tolerant training: Research how dstack can be used for fault-tolerant training of massive models.