Prompt Management, Tracing, and Evals: The New Table Stakes for GenAI Ops
From Data Engineering Podcast by Tobias Macey
February 15, 2026 · 51 min · Episode 501
About this episode
Aman Agarwal discusses the operational groundwork required for reliable and cost-effective LLM-powered applications.
Summary In this episode of the Data Engineering Podcast, Aman Agarwal, creator of OpenLit, discusses the operational groundwork required to run LLM-powered applications reliably and cost-effectively. He highlights common blind spots that teams face, including opaque model behavior, runaway token costs, and brittle prompt management, and explains how OpenTelemetry-native observability can turn these black-box interactions into stepwise, debuggable traces across models, tools, and data stores. Aman showcases OpenLit's approach to open standards, vendor-neutral integrations, and practical features such as fleet-managed OTEL collectors, zero-code Kubernetes instrumentation, prompt and secret management, and evaluation workflows. They also explore experimentation patterns, routing across models, and closing the loop from evals to prompt/dataset improvements, demonstrating how better visibility reshapes design choices from prototype to production. Aman shares lessons learned building in the open, where OpenLit fits and doesn't, and what's next in context management, security, and ecosystem integrations, providing resources and examples of multi-database observability deployments…
People in this episode
Host: Tobias Macey
Guest: Aman Agarwal
Topics covered
- GenAI Ops
- LLM-powered applications
- observability
- prompt management
- cost management
- data engineering
Keywords
- LLM
- OpenTelemetry
- observability
- prompt management
- cost-effective
- data stores
- Kubernetes
- evaluation workflows
Mentioned in this episode
Organizations: OpenLit, OpenTelemetry
More episodes of Data Engineering Podcast
- Text to Data Products: Kaarvi’s End-to-End AI for Ingestion, Quality, and Dashboards · June 8, 2026 · 53 min
- Scaling Graph Analytics Without ETL: Inside PuppyGraph’s Architecture · June 1, 2026 · 54 min
- Maximizing GPU Utilization: Heterogeneous Pipelines with Ray and Kubernetes · May 6, 2026 · 59 min
- The AI-First Data Engineer: 10–50x Productivity and What Changes Next · April 7, 2026 · 59 min
- Treat Metering Like Finance: Building Data Platforms for Consumption Economics · March 29, 2026 · 50 min
- Beyond the PDF: Rowan Cockett on Reproducible, Composable Science · March 22, 2026 · 43 min
Explore listener stats, chart rankings, contacts and more on the Data Engineering Podcast podcast page.