REFRAG with Xiaoqiang Lin - Weaviate Podcast #130!

About this episode

Xiaoqiang Lin discusses REFRAG, a method for improving LLM inference speed and context processing.

Xiaoqiang Lin is a Ph.D. student at the National University of Singapore. During his time at Meta, Xiaoqiang lead the research behind REFRAG: Rethinking RAG-based Decoding. Traditional RAG systems use vectors to retrieve relevant context with semantic search, but then throw away the vectors when passing the context to the LLM. REFRAG instead feeds the LLM these pre-compute vectors, achieving massive gains in long context processing and LLM inference speed! REFRAG makes Time-To-First-Token (TTFT) 31x faster and Time-To-Iterative-Token (TTIT) 3x faster, boosting overall LLM throughput by 7x while also being able to handle much longer contexts! There are so many interesting aspects to this and I really loved diving into the details with Xiaoqiang! I hope you enjoy the podcast!

People in this episode

Host: Weaviate

Guest: Xiaoqiang Lin

Topics covered

• RAG-based Decoding
• LLM inference
• semantic search
• context processing
• machine learning

Keywords

• REFRAG
• RAG systems
• LLM throughput
• context processing
• inference speed
• semantic search
• Time-To-First-Token
• Time-To-Iterative-Token

Mentioned in this episode

Organizations: National University of Singapore, Meta

Products: REFRAG

More episodes of Weaviate Podcast

• Knowledge Engineering with Bradley Allen - Weaviate Podcast #139! · June 1, 2026 · 1h 1m
• Booking.com and Weaviate with Başak Eskili - Weaviate Podcast #138! · May 18, 2026 · 42 min
• Search Agents with Nandan Thakur - Weaviate Podcast #137! · May 5, 2026 · 1h 1m
• AgentIR with Zijian Chen and Xueguang Ma - Weaviate Podcast #136! · April 27, 2026 · 1h 3m
• Data Agents with Shreya Shankar - Weaviate Podcast #135! · April 6, 2026 · 57 min
• Multi-Vector Search with Amélie Chatelain and Antoine Chaffin - Weaviate Podcast #134! · March 23, 2026 · 1h 21m