Kubernetes Bytes

In this episode of the Kubernetes Bytes podcast, Bhavin sits down with Kai-Hsun Chen, Software Engineer at Anyscale and maintainer of the KubeRay project. The discussion focuses on how the open source Ray project can help organizations use a single tool for data prep, model training, fine tuning and model serving workflows, both for their predictive AI and generative AI models. The discussion also dives into the KubeRay project and how it provides three different Kubernetes CRDs for Data Scientists to deploy Ray clusters on demand. Check out our website at https://kubernetesbytes.com/ Cloud Native News: * https://azure.github.io/AKS/2024/08/23/fine-tuning-language-models-with-kaito * https://orca.security/resources/blog/kubernetes-testing-environment/ * https://www.redhat.com/en/about/press-releases/red-hat-openstack-services-openshift-now-generally-available Show links: * Kai's LinkedIn: https://www.linkedin.com/in/kaihsun1996/ * KubeRay doc: https://docs.ray.io/en/latest/cluster/kubernetes/index.html * Ray Summit registration: https://raysummit.anyscale.com/flow/anyscale/raysummit2024/reg/createaccount (code: KaiHsunC15) * KubeRay repository: https://github.com/ray-project/kuberay * Ray repository: https://github.com/ray-project/ray * Ray Slack workspace: https://docs.google.com/forms/d/e/1FAIpQLSfAcoiLCHOguOm8e7Jnn-JJdZaCxPGjgVCvFijHB5PLaQLeig/viewform Timestamps: * 00:02:40 Cloud Native News * 00:07:20 Interview with Kai * 00:49:15 Key takeaways

In this episode of the Kubernetes Bytes podcast, Bhavin sits down with Sachi Desai, Product Manager and Paul Yu, Sr. Cloud Advocate at Microsoft to talk about the open source KAITO project. KAITO is the Kubernetes AI Toolchain Operator that enables AKS users to deploy open source LLM models on their Kubernetes clusters. They discuss how KAITO helps with running AI-enabled applications alongside the LLM models, how it helps users bring their own LLM models and run them as containers, and how KAITO helps them fine-tune open source LLMs on their Kubernetes clusters. Check out our website at https://kubernetesbytes.com/ Cloud Native News: * https://azure.github.io/AKS/2024/07/30/azure-container-storage-ga * https://github.blog/news-insights/product-news/introducing-github-models/ Show links: * Azure/kaito: Kubernetes AI Toolchain Operator - https://github.com/Azure/kaito/tree/main * https://www.youtube.com/watch?v=3cGmHDjR_3I&list=PLc3Ep462vVYtgN4rP1ThTJd2UlsBc2sou&index=2 * https://aka.ms/cloudnative/learnlive/intelligent-apps-on-aks/episode-2 * Jumpstart AI Workflows With Kubernetes AI Toolchain Operator - The New Stack - https://thenewstack.io/jumpstart-ai-workflows-with-kubernetes-ai-toolchain-operator * https://paulyu.dev/article/soaring-with-kaito/ * Concepts - Fine-tuning language models for AI and machine learning workflows - Azure Kubernetes Service | Microsoft Learn - https://learn.microsoft.com/en-us/azure/aks/concepts-fine-tune-language-models * Keep up to date on the most recent announcements by following some of the KAITO engineers on LinkedIn: 1. Fei Guo - https://www.linkedin.com/in/fei-guo-a48319a/ 2. Ishaan Sehgal - https://www.linkedin.com/in/ishaan-sehgal/ Timestamps: * 00:02:15 Cloud Native News * 00:05:34 Interview with Sachi and Paul * 00:42:08 Key takeaways

In this episode of the Kubernetes Bytes podcast, Bhavin sits down with Brandon Jacobs, an Infrastructure architect at Coreweave. They discuss how Coreweave has adopted Kubernetes to build the AI hyperscaler. The discussion dives into details around how Coreweave handles Day 0 and Day 2 operations for AI labs that need access to GPUs. They also talk about lessons learnt and best practices for building a Kubernetes based cloud. Check out our website at https://kubernetesbytes.com/ Episode Sponsor: Nethopper Learn more about KAOPS: nethopper.io For a supported-demo: info@nethopper.io Try the free version of KAOPS now! https://mynethopper.com/auth Cloud Native News: * https://siliconangle.com/2024/06/24/ollama-addresses-remote-execution-flaw-following-wiz-discovery/ * https://siliconangle.com/2024/06/18/suse-acquires-kubernetes-observability-startup-stackstate/ Show links: * https://www.linkedin.com/in/brandonrjacobs/ * https://www.coreweave.com/ Timestamps: * 00:01:39 Cloud Native News * 00:05:30 Interview with Brandon * 00:51:37 Key takeaways

In this episode of the Kubernetes Bytes podcast, Bhavin sits down with Bernie Wu, VP Strategic Partnerships and AI/CXL/Kubernetes Initiatives at Memverge. They discuss about how Kubernetes is the most popular platform to run AI model training and model inferencing jobs. The discussion dives into model training, talking about different phases of a DAG, and then talk about how Memverge can help users with efficient and cost-effective model checkpoints. The discussion goes into topics like saving costs by using spot instances, hot restart of training jobs, reclaiming unused GPU resources, etc. Check out our website at https://kubernetesbytes.com/ [https://www.youtube.com/redirect?event=video_description&redir_token=QUFFLUhqbllTN0VpRHpuZWZiem9iWWc2bDF5NF9SMGhod3xBQ3Jtc0tuMnNGTmhwN01xdC01ZEFqRjFERnZBLUlUSTFCRWVta0tYQU4wWndITWhfV3RnbHh1cFVtbUc5NGZRandUcG1ocjQ5Q2R2cG1tQmpibkhJMVNRMHIycE1SYVJNdlVWMGFCa2xwTkhIOEFZUFRqVG5sdw&q=https%3A%2F%2Fkubernetesbytes.com%2F&v=Jf059PFn6l0] Episode Sponsor: Nethopper * Learn more about KAOPS: @nethopper.io * For a supported-demo: info@nethopper.io * Try the free version of KAOPS now! https://mynethopper.com/auth [https://www.youtube.com/redirect?event=video_description&redir_token=QUFFLUhqa2c2UWw4bFZISFN2X0hIWUtuWHlpWGk3WC1RZ3xBQ3Jtc0trQ3FpMlZDeXUwcVdZZGJ1d2R5RVREc1V0SzNPeXBzZGVCeWdzYjNsRndTVzRGcTYtUVFRUnI2bXEtWE96QVpNNThCaENiWXpqM3dzSmF6cml1c1BEYUk4bnlEQ3lqc1d0cXdTVWxuVE5YZWV3elNITQ&q=https%3A%2F%2Fmynethopper.com%2Fauth&v=Jf059PFn6l0] Cloud Native News: * https://www.aquasec.com/blog/linguistic-lumberjack-understanding-cve-2024-4323-in-fluent-bit/ * https://kubernetes.io/blog/2024/05/20/completing-cloud-provider-migration/ * https://thenewstack.io/introducing-aks-automatic-managed-kubernetes-for-developers/ * https://www.harness.io/blog/harness-to-acquire-split Show Links: * https://www.linkedin.com/in/berniewu/ * https://criu.org/Main_Page * https://memverge.com/ * https://youtu.be/tY8YOMRuqWI?si=yB3hHqLUpYPZ-KWN * https://youtu.be/ND4seSKpJHI?si=shh0iuA9qC-dO6eb Timestamps: * 01:04 [https://www.youtube.com/watch?v=Jf059PFn6l0&t=64s] Cloud Native News * 08:47 [https://www.youtube.com/watch?v=Jf059PFn6l0&t=527s] Interview with Bernie * 51:40 [https://www.youtube.com/watch?v=Jf059PFn6l0&t=3100s] Key takeaways

In this episode of the Kubernetes Bytes podcast, Ryan and Bhavin talk to Torsten Steinbach - VP, Chief Architect for Analytics & AI at EDB about all things Vector Databases, Postgres, and why Data is important for building AI platforms. The discussion dives into how vector databases are different than relational databases and why using Postgres extensions helps organizations use their existing data for AI applications. Check out our website at https://kubernetesbytes.com/ Kubernetes Community Days (KCD) in New York City on May 22nd, use the promo code "KUBERNETESBYTES" to get a 10% discount on your registration fees! Episode Sponsor: Nethopper * Learn more about KAOPS: @nethopper.io * For a supported-demo: info@nethopper.io * Try the free version of KAOPS now! * https://mynethopper.com/auth Cloud Native News: * https://www.reuters.com/markets/deals/ibm-nearing-buyout-deal-hashicorp-wsj-reports-2024-04-23/ * https://www.wiz.io/blog/wiz-acquires-gem-security-to-reinvent-threat-detection-in-the-cloud * https://techcrunch.com/2024/04/18/wiz-is-in-talks-to-buy-lacework-for-150-200m-security-firm-was-last-valued-at-8-3b/ * https://www.prnewswire.com/news-releases/coreweave-secures-1-1-billion-in-series-c-funding-to-drive-the-next-generation-of-cloud-computing-for-the-future-of-ai-302133328.html * https://kubernetes.io/blog/2024/04/17/kubernetes-v1-30-release * https://dok.community/blog/become-a-data-on-kubernetes-in-2024-ambassador/ Show Links: * https://www.enterprisedb.com/news/edb-acquires-splitgraph * https://www.enterprisedb.com/resources/events * https://www.enterprisedb.com/ Timestamps: * 00:03:22 Cloud Native News * 00:17:45 Interview with Torsten * 00:57:00 Key takeaways

In this episode of the Kubernetes Bytes podcast, Ryan and Bhavin talk to Charlie Egan, Sr. Developer Advocate at Styra about all things Open Policy Agent or OPA. This episode is meant to be a 101 level episode, where we will learn what OPA is and how it can help improve the security posture for your applications running on Kubernetes. The discussion dives into some of the use cases for OPA and how it helps with Kubernetes admission control, auditing, etc. Check out our website at https://kubernetesbytes.com/ Episode Sponsor: Elotl Links: * https://elotl.co/luna * https://www.elotl.co/luna-free-trial Timestamps: * 07:46 Interview with Charlie * 01:02:00 Key takeaways Show Links: * OPA Ecosystem https://www.openpolicyagent.org/ecosystem/ * OPA Envoy Integration https://github.com/open-policy-agent/opa-envoy-plugin * Charlie's site: https://charlieegan3.com * OPA Docs: https://www.openpolicyagent.org/docs/latest/ * OPA Playground: https://play.openpolicyagent.org * OPA Slack: https://communityinviter.com/apps/openpolicyagent/signup

In this episode of the Kubernetes Bytes podcast, Ryan and Bhavin sit down with Janakiram MSV - an advisor, analyst and architect to talk about how users can run Generative AI models on Kubernetes. The discussion revolves around Jani's home lab and his experimentation with different LLM models and how to get them running on NVIDIA GPUs. Jani has spent the past year becoming a subject matter expert in GenAI, and this discussion highlights all the different challenges he faced and what lessons he learnt from them. Check out our website at https://kubernetesbytes.com/ Episode Sponsor: Elotl * https://elotl.co/luna * https://www.elotl.co/luna-free-trial Timestamps: * 02:02 Cloud Native News * 15:31 Interview with Jani * 01:11:00 Key takeaways Cloud Native News: * https://www.techerati.com/press-release/octopus-deploy-acquires-codefresh-to-boost-kubernetes-and-cloud-native-delivery/ * https://www.civo.com/blog/kubefirst-joins-civo * https://cast.ai/kubernetes-cost-benchmark * https://www.techradar.com/pro/vmware-customers-are-jumping-ship-as-broadcom-sales-continue-heres-where-theyre-moving-to * https://cloudonair.withgoogle.com/events/techbyte-making-ai-ml-scalable-cost-effective-gke * https://dok.community/dok-events/dok-day-kubecon-paris/ * https://training.linuxfoundation.org/certification/certified-argo-project-associate-capa Show Links: * https://www.youtube.com/janakirammsv * https://www.linkedin.com/in/janakiramm/ * - NVIDIA Container Toolkit - https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/index.html * NVIDIA Device Plugin - https://github.com/NVIDIA/k8s-device-plugin * NVIDIA Feature Discovery - https://github.com/NVIDIA/gpu-feature-discovery * Hugging Face Text Gen Inference - https://huggingface.co/docs/text-generation-inference/index * Hugging Face Text Embeddings Inference - https://huggingface.co/docs/text-embeddings-inference/index * ChromaDB - https://www.trychroma.com/

In this episode of the Kubernetes Bytes podcast, Bhavin sits down with Jason Dobies - Director of Edge Engineering at SUSE to talk about all things K3s. They discuss why Kubernetes is best suited for Edge deployments, and why K3s was built and how it helps users architect their edge solutions. The discussion goes into topics like Security, Storage, High Availability at the Edge. Check out our website at https://kubernetesbytes.com/ Episode Sponsor: Elotl * https://elotl.co/luna * https://www.elotl.co/luna-free-trial Timestamps: * 01:20 Cloud Native News * 06:01 Interview with Jason * 50:42 Key takeaways Cloud Native News: * https://vmblog.com/archive/2024/01/29/dynatrace-to-acquire-runecast-to-enhance-cloud-native-security-and-compliance.aspx - * https://chronosphere.io/news/chronosphere-acquires-calyptia/ Show links: * https://k3s.io/ * https://www.linkedin.com/in/jdob/