I've been working on a little open source [1] search engine, nilch. I noticed that nearly all well known search engines, including the alternative ones, tend to be run by companies of various sizes with the goal to make money, so they either fill your results with ads or charge you money, and I dislike this because search is the backbone of the internet and should not be commercial, so it runs in a not-for-profit style and aims to survive on donations. Additionally I'm personally really sick of AI in my search results so I got rid of that, and I wanted DuckDuckGo bangs so it supports all of them. Like many alternative search engines, it is fully private.
Sadly, it currently does not have its own index but rather uses the Brave search API. Once I'm in a financial position that it's possible, I would absolutely love to build a completely new index from the ground up which is open source, as well as an open source ranking and search algorithm, to back it.
I posted on Reddit and got an amazing amount of feedback which I implemented a number of feature requests, so I would really like your ideas, critiques, and bug reports as well. Thank you and sorry for the long post!
[1] https://github.com/UnmappedStack/nilch
Imustaskforhelp•2h ago
There was this project on hackernews which was recently shown where they (scraped?) the internet and then created an really efficient embedding of the search engine. I wish if you could look more into it or contact the creator of that project perhaps.
https://news.ycombinator.com/item?id=44878151 (Show HN: Building a web search engine from scratch with 3B neural embeddings)
Looks like https://search.wilsonl.in/ they have since then closed the live demo but I had used it when it was live and in my opinion, it was a mix of that if things needed some improvements but that it was also usable for some things which were in the dataset (Of course you wouldn't get Organic chemistry questions/answers for high schoolers as an example in there but you will find most things (usually wikipedia) and then some good sources, usually the ones popular but it was really cool overall so perhaps you can look more into it and helps
Now I really love your project a lot and I think there should be not for profit search engines, but I am a little worried about using it since if I use it as my search engine, then it might cost you a lot of money (using the brave api) .
I just searched and it seems that ecosia is a non profit as well so you can definitely partner up with them, I remember a post about qwant and ecosia partnering up to create an independent search engine.
I think that there should be competition within the search engine space especially via non profits in a way similar to wikipedia one might say ideally. Wishing you the best for this project's future!
prmoustache•2h ago
UnmappedStack•2h ago
UnmappedStack•2h ago
I'll have a look into that project, thank you. Cost is a slight issue so far, yes. There have been about 4,000 searches in the past couple days but I've slightly improved cost efficiency with caching, and I've received two small donations which do help a bit, so the hope is that donations will be able to sustain it.
Partnering with Ecosia is a really interesting idea, however I think that there may be a conflict of interest since they do aim to make money with ads, just to go towards environmental efforts rather than a corporation. They would be disadvantaged if nilch was at an advantage over their users.
I do love the wikipedia model and I hope that nilch can run similarly. Thank you again!