Consider this simple test I’ve been running:

Anchor: “A background service listens to a task queue and processes incoming data payloads using a custom rules engine before persisting output to a local SQLite database.”

Option A (Lexical Match): “A background service listens to a message queue and processes outgoing authentication tokens using a custom hash function before transmitting output to a local SQLite database.”

Option B (Semantic Match): “An asynchronous worker fetches jobs from a scheduling channel, transforms each record according to a user-defined logic system, and saves the results to an embedded relational data store on disk.”

Any decent LLM (e.g., Gemini 2.5 Pro, GPT-4/5) immediately knows that the Anchor and Option B describe the same concept just with different words. But when I test embedding models like gemini-embedding-001 (currently top of MTEB), they consistently rate Option A as more similar measured by cosine similarity. They’re getting tricked by surface-level word overlap.

I put together a small GitHub repo that uses ChatGPT to generate and test these “semantic triplets:

https://github.com/semvec/embedstresstest

gemini-embedding-001 (current #1 on MTEB leaderboard ) scored close to 0% on these adversarial examples.

The repo is unpolished at the moment but it gets the idea across and everything is reproducible.

Anyway, did anyone else notice this problem?