Longtime lurker and really happy to be writing this post. I'm excited to share a proof of concept I've been working on for efficient vector database distribution called Ragged. In my paper and PoC, I explore leveraging the MP4 video container format to store and distribute high-dimensional vectors for semantic search applications.
The idea behind Ragged is to encode vectors and their metadata into MP4 files using custom tracks, allowing seamless distribution through existing Content Delivery Networks (CDNs). This approach maintains compatibility with standard video infrastructure while achieving comparable search performance to traditional vector databases.
Key highlights of my work include: - A novel encoding scheme for high-dimensional vectors and metadata into MP4 container formats. - CDN-optimized architecture with HTTP range requests, fragment-based access patterns, and intelligent prefetching. - Comprehensive evaluation showing significant improvements in cold-start latency and global accessibility. - An open-source implementation to facilitate reproduction and adoption.
I was inspired by the innovative work of Memvid (https://github.com/Olow304/memvid), which demonstrated the potential of using video formats for data storage. My project builds on this concept with a focus on CDNs and semantic search.
I believe Ragged offers a promising solution for deploying semantic search capabilities in edge computing and serverless environments, leveraging the mature video distribution ecosystem. Also sharing indexed knowledge bases in the form of offline MP4 can unlock a new class of applications.
I'm eager to hear your thoughts, feedback, and any potential use cases you envision for this approach. You can find the full paper and implementation details [here](https://github.com/nikitph/ragged).
Thank you for your time fellows
Nikit