Roughly, first there is the query analysis/manipulation phase where you might have NER, spell check, query expansion/relaxation etc

Then there is the selection phase, where you retrieve all items that are relevant. Sometimes people will bring in results from both text and vector based indices. Perhaps and additional layer to group results

Then finally you have the reranking layer using a cross encoder model which might even have some personalisation in the mix

Also, with vector search you might not need query expansion necessarily since semantic similarity does loose association. But every domain is unique and there’s only one way to find out

Embeddings are huge compared to what you need with FTS, which generally has good locality, compresses extremely well, and permits sub-linear intersection algorithms and other tricks to make the most of your IOPS.

Regardless of vector size, you are unlikely to get more than one embedding per I/O operation with a vector approach. Even if you can fit more vectors into a block, there is no good way of arranging them to ensure efficient locality like you can with e.g. a postings list.

Thus off a 500K IOPS drive, given a 100ms execution window, your theoretical upper bound is 50K embeddings ranked, assuming actual ranking takes no time and no other disk operations are performed and you have only a single user.

Given you are more than likely comparing multiple embeddings per document, this carriage turns to a pumpkin pretty rapidly.

Information about how Bing text search works appears to be pretty sparse though.

One of the great mysteries to me right now is how ChatGPT search actually works.

It was Bing when they first launched it, but OpenAI have been investing a ton into their own search infrastructure since then. I can't figure out how much of it is Bing these days vs their own home-rolled system.

What's confusing is how secretive OpenAI are about it! I would personally value it a whole lot more if I understood how it works.

So maybe it's way more vector-based than I believe.

I'd expect any modern search engine to have aspects of vectors somewhere - some kind of hybrid BM25 + vectors thing, or using vectors for re-ranking after retrieving likely matches via FTS. That's different from being pure vectors though.