> Google must provide Web Search Index data (URLs, crawl metadata, spam scores) at marginal cost.
Maybe they're shutting down the good integration and then Kagi, Ecosia and others can buy index data in an inconvenient way going forward?
Kagi makes deals with many search engines so they can have raw search results in exchange for money.
Google says: no, you can't have raw search results because only whales can get those. Only thing we can offer you is search results riddled with ads and we won't allow you to reorder or filter them.
Kagi thinks Google's offer is unacceptable, so Kagi goes to a third party SERP API, which scrapes Google at scale and sells the raw search results to Kagi and others.
August 2024: Court says Google is breaking the law by selling raw search results only to whales.
December 2025: Court orders that for the next six years, 1. Google must no longer exclude non-whales from buying raw search results, 2. Google must offer the raw search results for a reasonable price, and 3. Google can no longer force partners to bundle the results with ads.
December 2025: Google sues the third-party scraping companies.
January 2026: Google says "hey, the old search offering is going to go away, there's going to be a new API by 2027, stay tuned."
The correct parsing is: "Google is ending (full-web search for niche search engines)"
What they are ending is their support for websites to search across the entire web. The websites that search across the entire web are usually niche search engine websites.
They were:
> aiming to serve 30% of French search queries [by end of 2025]
https://blog.ecosia.org/launching-our-european-search-index/
PageRank wouldn't exist without webrings, directories, and forums you could only search individually, and we thrived on that Internet.
Welcome back, ye olde Internet.
> The French index is at an advanced stage of completion, we have started creating the German language index, and the English one should start shortly. All progress is quickly integrated into the Qwant STAAN API.
But of course they managed to cut themselves a nice salary with EU funds, paid in part by me and you, so that's all that matters.
I found it didn’t really work as a real search engine but it was interesting.
Hard part is doing it at any sort of scale and producing useful results. It's easy to build something that indexes a few million documents. Pushing into billions is a bigger challenge, as you start needing a lot of increasingly intricate bespoke solutions.
Devlog here:
https://www.marginalia.nu/tags/search-engine/
And search engine itself:
https://marginalia-search.com/
(... though it operates a bit sub-optimally now as I'm using a ton of CPU cores to migrate the index to use postings lists compression, will take about 4-5 days I think).
AMD EPYC 7543 x2 for 64 cores/128 threads
512 GB RAM
~ 90 TB of PM9A3 SSDs across 12 physical devices
Storage is not very full though. I'm probably using about a third of it at this point.
I suppose I'm asking whether this is actually a _good thing_ in that it will stimulate competition in the space, or if it's just a case that Google's index is now too good for anyone to reasonably catch up at this point.
Bing charge per query for the average user. Ecosia and Qwant use Bing to power their results, probably under some type of license, which results in them paying much less per query than a normal user.
If you want programmatic access to search results there aren't really many options left.
But for searching in more niche languages google is usually the only decent option and I have little hope that others will ever reach the scale where they could compete.
What alternatives are there besides Bing? Is it really so hard that it’s not considered worth doing? Some of the AI companies (Perplexity, Anthropic) seem to have managed to get their own indexing up and running.
Just dissolve them in acid.
If you actually enforce them.
Unfortunately, during the Reagan administration, political sentiment toward monopolies shifted and since then antitrust law has been a paper tiger at best.
01jonny01•2h ago
If you actually need whole-web search, Google now points you to an “interest form” for enterprise solutions (Vertex AI Search etc.), with no public pricing and no guarantee they’ll even reply.
This seems like it effectively ends the era of indie / niche search engines being able to build on Google’s index. Anything that looks like general web search is getting pushed behind enterprise gates.
I haven’t seen much discussion about this yet, but for anyone who built a small search product on Programmable Search, this feels like a pretty big shift.
Curious if others here are affected or already planning alternatives.
UPDATE: I logged into Programmable Search and the message is even more explicit: Full web search via the "Search the entire web" feature will be discontinued within the next year. Please update your search engine to specify specific sites to search. With this link: https://support.google.com/programmable-search/answer/123971...
throwaway_20357•2h ago
doublerabbit•2h ago
marginalia_nu•1h ago
https://blog.kagi.com/waiting-dawn-search
Which saw some discussion on HN.
embedding-shape•1h ago
~450 score, ~247 comments and still on /best ("Most-upvoted stories of the last 48 hours"):
https://news.ycombinator.com/item?id=46708678 - "Waiting for dawn in search: Search index, Google rulings and impact on Kagi"
pell•1h ago
echelon•1h ago
Google is a monopoly across several broad categories. They're also a taxation enterprise.
Google Search took over as the URL bar for 91% of all web users across all devices.
Since this intercepts trademarks and brand names, Google gets to tax all businesses unfairly.
Tell your legislators in the US and the EU that Google shouldn't be able to sell ads against registered trademarks (+/- some edit distance). They re-engineered the web to be a taxation system for all businesses across all categories.
Searching for Claude -> Ads in first place
Searching for ChatGPT -> Ads in first place
Searching for iPhone -> Ads in first place
This is inexcusable.
Only searches for "ChatGPT versus", "iPhone reviews", or "Nintendo game comparison" should allow ads. And one could argue that the "URL Bar" shouldn't auto suggest these either when a trademark is in the URL bar.
If Google won't play fair, we have to kill 50% of their search revenue for being egregiously evil.
If you own a trademark, Google shouldn't be able to sell ads against you.
--
Google's really bad. Ideally we'd get an antitrust breakup. They're worse than Ma Bell. I wouldn't even split Google into multiple companies by division - I'd force them to be multiple copies of the same exact entity that then have to compete with each other:
Bell Systems -> {BellSouth, Bell Atlantic, Southwestern Bell, ...}
Google -> {GoogleA, GoogleB, GoogleC, ...}
They'd each have cloud, search, browser, and YouTube. But new brand names for new parent companies. That would create all-out war and lead to incredible consumer wins.
marginalia_nu•1h ago
[1] https://en.wikipedia.org/wiki/Essential_facilities_doctrine
nemosaltat•1h ago
[0]> Google: Google does not offer a public search API. The only available path is an ad-syndication bundle with no changes to result presentation - the model Startpage uses. Ad syndication is a non-starter for Kagi’s ad-free subscription model.[^1]
[0]> The current interim approach (current as of Jan 21, 2026)
[0]> Because direct licensing isn’t available to us on compatible terms, we - like many others - use third-party API providers for SERP-style results (SERP meaning search engine results page). These providers serve major enterprises (according to their websites) including Nvidia, Adobe, Samsung, Stanford, DeepMind, Uber, and the United Nations.
I’m an avid Kagi user, and it seems like Kagi and some other notable interested parties have _already_ been unable to do get what they want/need with Google’s index.
[0]> The fact that we - and companies like Stanford, Nvidia, Adobe, and the United Nations - have had to rely on third-party vendors is a symptom of the closed ecosystem, not a preference.
Hopefully someone here can clarify for me, or enumerate some of these “third-party vendors” who seem like they will/might/could be directly affected by this.
[0] antibabelic > relevant https://blog.kagi.com/waiting-dawn-search [1] https://blog.kagi.com/dawn-new-era-search > [^1]: A note on Google’s existing APIs: Google offers PSE, designed for adding search boxes to websites. It can return web results, but with reduced scope and terms tailored for that narrow use case. More recently, Google offers Grounding with Google Search through Vertex AI, intended for grounding LLM responses. Neither is general-purpose index access. Programmable Search Engine is not designed for building competitive search. Grounding with Google Search is priced at $35 per 1,000 requests - economically unviable for search at scale, and structured as an AI add-on rather than standalone index syndication. These are not the FRAND terms the market needs
tpetry•39m ago
That must be the reason why they limit the searches you can do in the starter plan. Every SerpApi call costs money.
saltysalt•1h ago
People rely too much on other people's infra and services, which can be decommissioned anytime. The Google Graveyard is real.
orf•1h ago
I just searched for “stackoverflow” and the first result was this: https://www.perl.com/tags/stackoverflow/
The actual Stackoverflow site was ranked way down, below some weird twitter accounts.
saltysalt•1h ago
orf•1h ago
It’s cool though, and really fast
saltysalt•1h ago
direwolf20•45m ago
saltysalt•25m ago
You are absolutely right, it is the hardest part!
globular-toast•36m ago
I feel like Google-style "search" has made people really dumb and unable to help themselves.
orf•20m ago
dredmorbius•27m ago
Even with PageRank result prioritisation is highly subject to gaming. Raw keyword search is far more so (keyword stuffing and other shenanigans), moreso as any given search engine begins to become popular and catch the attention of publishers.
Google now applies other additional ordering factors as well. And of course has come to dominate SERP results with paid, advertised, listings, which are all but impossible to discern from "organic" search results.
(I've not used Google Web Search as my primary tool for well over a decade, and probably only run a few searches per month. DDG is my primary, though I'll look at a few others including Kagi and Marginalia, though those rarely.)
<https://en.wikipedia.org/wiki/PageRank>
"The anatomy of a large-scale hypertextual Web search engine" (1998) <http://infolab.stanford.edu/pub/papers/google.pdf> (PDF)
Early (1990s) search engines: <https://en.wikipedia.org/wiki/Search_engine#1990s:_Birth_of_...>.
renegat0x0•1h ago
https://github.com/rumca-js/Internet-Places-Database
Demo for most important ones https://rumca-js.github.io/search
saltysalt•1h ago
johnofthesea•1h ago
saltysalt•1h ago
lolive•29m ago
salawat•1h ago
I can only weep at this point, as the heroes that were the Silent and Greatest generations (in the U.S.), who fought hard to pass on as much institutional knowledge as possible through hardcore organization and distribution via public and University library, have had that legacy shit on by these ad obsessed cretins. The entirety of human published understanding; and we make it nigh impossible for all but the most determined to actually avail themselves of it.
raincole•54m ago
TIL they allowed that before. It sounds a bit crazy. Like Google is inviting people to repackage google search itself and sell it / serve with their own ads.