Don't miss out! Join us at our next Flagship Conference: KubeCon + CloudNativeCon North America in Salt Lake City from November 12 - 15, 2024. Connect with our current graduated, incubating, and sandbox projects as the community gathers to further the education and advancement of cloud native computing. Learn more at https://kubecon.io

Self-Hosted LLMs on Kubernetes: A Practical Guide - Hema Veeradhi & Aakanksha Duggal, Red Hat

Have you ever considered deploying your own large language model (LLM), but the seemingly complex process held you back from exploring this possibility? The complexities of deploying and managing LLMs in production environments often pose significant challenges. This talk will serve as a comprehensive introductory guide, empowering beginners to commence their LLM journey by effectively hosting their own models on Kubernetes. We will discuss the process of selecting appropriate open source LLM models, containerization of the models, and creating Kubernetes deployment manifests and resource provisioning to support the LLM's computational needs. Self-hosted LLMs offer enhanced data privacy, flexibility in model training, and reduced operational costs, making them an attractive option for organizations seeking greater control over their AI infrastructure. By the end of this talk, attendees will possess the necessary skills and knowledge to navigate the exciting path of self-hosting LLMs.