From Blind Spots to Observability: Operationalizing LLM Apps with OpenLit

About this episode

Aman Agarwal discusses operationalizing LLM-powered applications with a focus on observability and cost management using OpenLit.

Summary  In this episode of the AI Engineering Podcast, Aman Agarwal, creator of OpenLit, discusses the operational foundations required to run LLM-powered applications in production. He highlights common early blind spots teams face, including opaque model behavior, runaway token costs, and brittle prompt management, emphasizing that strong observability and cost tracking must be established before an MVP ships. Aman explains how OpenLit leverages OpenTelemetry for vendor-neutral tracing across models, tools, and data stores, and introduces features such as prompt and secret management with versioning, evaluation workflows (including LLM-as-a-judge), and fleet management for OpenTelemetry collectors. The conversation covers experimentation patterns, strategies to avoid vendor lock-in, and how detailed stepwise traces reshape system design and debugging. Aman also shares recent advancements like a Kubernetes operator for zero-code instrumentation, multi-database configurations for environment isolation, and integrations with platforms such as Grafana and Dash0. They conclude by discussing lessons learned from building in the open, prioritizing reliability, developer…

People in this episode

Host: Tobias Macey

Guest: Aman Agarwal

Topics covered

• LLM applications
• observability
• cost tracking
• prompt management
• OpenTelemetry
• Kubernetes
• data security

Keywords

• LLM
• OpenLit
• observability
• cost tracking
• OpenTelemetry
• Kubernetes
• prompt management
• data security
• vendor lock-in
• system design

Mentioned in this episode

Organizations: OpenLit, OpenTelemetry, Grafana, Dash0

More episodes of AI Engineering Podcast

• Kubernetes, Compliance, and Control: The Operational Backbone of AI Sovereignty · February 25, 2026 · 1h 1m
• Taming Voice Complexity with Dynamic Ensembles at Modulate · February 8, 2026 · 59 min
• GPU Clouds, Aggregators, and the New Economics of AI Compute · January 27, 2026 · 46 min
• The Future of Dev Experience: Spotify’s Playbook for Organization‑Scale AI · January 20, 2026 · 56 min
• Generative AI Meets Accessibility: Benchmarks, Breakthroughs, and Blind Spots with Joe Devon · January 5, 2026 · 56 min
• Beyond the Chatbot: Practical Frameworks for Agentic Capabilities in SaaS · December 29, 2025 · 54 min