Autoscaling model inference for traditional ML/AI workloads is a relatively well-understood problem; these workloads are typically scaled horizontally, adding extra containers under increased load. But what do you do when you have millions of requests and larger models that span multiple machines and GPUs, requiring significant workload coordination?

In this talk, Federico will take the perspective of an MLOps team tasked with reducing the AI vendor spending of a large enterprise, deploying increasingly large open-source language models on Kubernetes. He will identify a series of practical inference and system architecture best practices that teams can follow to autoscale model serving in their clusters.

Furthermore, Federico will take a deep dive into components of the open-source ecosystem that teams can deploy to optimize the utilization of their compute (GPU) resources and increase the number of tokens per second (TPS) served.

Talk Structure

• Quick intro to autoscaling methods on Kubernetes and language model inference
• Explanation of the demo task to the audience
• Deep dive into autoscaling model serving on Kubernetes, starting with smaller single-container models, then moving to large models that span multiple containers, and finally adding GPU complexity
• Particularities of autoscaling GPU resources
• Open-source tools that can simplify “some” of this work (Ray, vLLM, SGLang, etc.)

Conclusion
This is a very interactive talk with multiple questions for the audience throughout the presentation.