When Jev launched last month I assumed it was a novelty. Then we tried it on the job our AI receptionist finds hardest. A caller asks something oddly specific, like whether a free valuation ties them into selling with the agent, and the answer is buried somewhere on the business’s website.
We gave Jev a one-line description of every page on the site and asked it one choice question, which page answers this, with “none of these” as the last option. Across four of our customers’ sites (22 to 232 pages) and 168 caller questions, it put the right page first 88% of the time, against 70% for our production embeddings. The number I care about more is how often the LLM got handed the wrong page, 3% against 33%. When the site didn’t have the answer, Jev said so 95% of the time and our current search 23%. Each lookup took about a third of a second, and all 1,449 requests cost about 45 cents.
Fair warning, this piece is written for vibe coders and tech business owners rather than engineers, so it’s the plain English version. The method, charts and the bits that went wrong are in the companion write-up: https://sayora.ai/many-hands/jev-vs-embeddings/
Most of the failures were ours. Jev reads page descriptions literally. A plumber’s installation page mentioned water softeners in the body but not in its description, so Jev said “none of these” at 96%. Above 255 options or about 64k tokens you need embeddings to shortlist first. And we couldn’t find a published data processing agreement (DPA), which matters with UK callers’ data.
Happy to answer anything about the setup or where it didn’t work.
Bocajmai•54m ago
We gave Jev a one-line description of every page on the site and asked it one choice question, which page answers this, with “none of these” as the last option. Across four of our customers’ sites (22 to 232 pages) and 168 caller questions, it put the right page first 88% of the time, against 70% for our production embeddings. The number I care about more is how often the LLM got handed the wrong page, 3% against 33%. When the site didn’t have the answer, Jev said so 95% of the time and our current search 23%. Each lookup took about a third of a second, and all 1,449 requests cost about 45 cents.
Fair warning, this piece is written for vibe coders and tech business owners rather than engineers, so it’s the plain English version. The method, charts and the bits that went wrong are in the companion write-up: https://sayora.ai/many-hands/jev-vs-embeddings/
Most of the failures were ours. Jev reads page descriptions literally. A plumber’s installation page mentioned water softeners in the body but not in its description, so Jev said “none of these” at 96%. Above 255 options or about 64k tokens you need embeddings to shortlist first. And we couldn’t find a published data processing agreement (DPA), which matters with UK callers’ data.
Happy to answer anything about the setup or where it didn’t work.