I am SUPER EXCITED to publish the 130th episode of the Weaviate Podcast featuring Xiaoqiang Lin, a Ph.D. student at the National University of Singapore! During his time at Meta, Xiaoqiang lead the research behind REFRAG: Rethinking RAG-based Decoding!

Traditional RAG systems use vectors to find relevant contexts with semantic search, but then throw away these vectors when it is time to pass the retrieved information to the LLM! REFRAG instead feeds the LLM these pre-computed vectors, achieving massive gains in long context processing and LLM inference speeds!

REFRAG makes Time-To-First-Token (TTFT) 31x faster and Time-To-Iterative-Token (TTIT) 3x faster, boosting overall LLM throughput by 7x while also being able to handle much longer contexts!

This is such an exciting evolution for the applications of Vector Databases, and Weaviate’s mission to weave AI and Database systems together! I loved diving into the details of REFRAG with Xiaoqiang, I hope you enjoy the podcast!

YouTube: https://www.youtube.com/watch?v=yi7v-UXMg0U

Spotify: https://spotifycreators-web.app.link/e/RWvmvMgRZXb