Everyone has their own challenge for generative AI. Mine was a search engine for about €150. A month or so later, here it is. Claude Code wrote most of the code; the architecture and storage layer are mine, built on years of earlier work. €150 is compute and storage at hetzner, and one month of claude pro max x5.
It needs polish here and there, but the results hold up, and some of its quirks surprised me in good ways. No ads, no tracking, no personalisation.
The corpus:
English (93M docs, 2.1 TB): Stack Overflow, arXiv, Wikipedia, Hacker News and Lobsters up to 2020
French (18M docs, 0.5 TB): theses.fr, Persée, Gallica, Wikisource, Wikipédia
How it’s built, since that’s probably the interesting part here:
Everything is Chez Scheme. The storage layer is an ordered key-value store (it implements SRFI-167/168, which I wrote) running as an LSM tree.
The C parts: libtls, liburing, picohttpparser, libc.
French and English are separate indexes. Mixing them corrupted document frequencies in an earlier version, because French and English words collide after stemming.
Known weaknesses: ranking is the next big piece of work.
What would help most: a query where it failed you, or a source you think belongs in the corpus and why. I’m especially curious which corpora people would put in a “technical history” index.
The code isn’t public yet. I want to see what people actually use before I freeze the design.
The way you present the search results should be much better. Previews, etc.
amirouche•52m ago
Fair. The goal is eventually to convert to paid search for people with their own harness, or deep search. Preview or not, does not change the balance, when what you need is depth.
amirouche•1h ago
It needs polish here and there, but the results hold up, and some of its quirks surprised me in good ways. No ads, no tracking, no personalisation.
The corpus:
English (93M docs, 2.1 TB): Stack Overflow, arXiv, Wikipedia, Hacker News and Lobsters up to 2020
French (18M docs, 0.5 TB): theses.fr, Persée, Gallica, Wikisource, Wikipédia
How it’s built, since that’s probably the interesting part here:
Everything is Chez Scheme. The storage layer is an ordered key-value store (it implements SRFI-167/168, which I wrote) running as an LSM tree.
The C parts: libtls, liburing, picohttpparser, libc. French and English are separate indexes. Mixing them corrupted document frequencies in an earlier version, because French and English words collide after stemming.
Known weaknesses: ranking is the next big piece of work.
What would help most: a query where it failed you, or a source you think belongs in the corpus and why. I’m especially curious which corpora people would put in a “technical history” index.
The code isn’t public yet. I want to see what people actually use before I freeze the design.
There is json output: https://h9p3r.fr/?q=biological+intelligence+future+&lang=en&...