Why does this work so well?

How big were the original PDFs? Are they just text or images and other formatting too?

This is an extremely bad method of storing text data. Video codecs are not particularly efficient at compressing QR codes, given the high contrast between the blocks defeating the traditional DCT psychovisual assumptions of smooth gradients. There is little to no redundancy between QR code encodings of similar text.

You'd probably have a smaller database and better results crunching text into a zip file, or compressed rows in a sqlite database, or any other simple random-access format.

Is this more efficient than putting all of that in say a 7z archive?

I'd expect video frames to be maximally efficient if you sorted the chunks by image similarity somehow.

Also isn't there a risk of losing data by doing this since for example h.265 is lossy?

Why not just do it locally? Or were the RAM consumption and the cloud cost comments distinct?

> The results were surprising. 10,000 PDFs compressed down to a 1.4GB video file.

And how big was the total text in those PDFs?

> Video compression handles redundancy between similar documents effectively.

Definitely not. None of the "redundancy" between, or within, texts (e.g. repeated phrases) is apparent in a sequence of images of QR codes.

Cut the cloud vendors out of the picture and build and query your index on a spare linux box.

I've only played with TF-IDF/BM25 as opposed to vector searches, but there's no way your queries should be taking so long on such a small corpus. Querying 10k documents feels like 2-10ms territory, not 900ms.

April Fools?

900 ms sounds like a lot for just 10,000 documents? How many chunks are there per document? Maybe Pinecone's 820 ms includes network latency plus they need to serve other users?

In Go, I once implemented a naive brute-force cosine search (linear scan in memory), and for 1 million 350-dimensional vectors, I got results in under 1 second too IIRC.

I ended up just setting up OpenSearch, which gives you hybrid semantic + full-text search out of the box (BM25 + kNN). In my tests, it gave better results than semantic search alone, something like +15% better retrieval.

I’m not sure why this is getting so much hate. This could be groundbreaking.

can you provide that mp4