Web-scraping AI bots cause disruption for scientific databases and journals

https://www.nature.com/articles/d41586-025-01661-4

30•tchalla•1d ago

Comments

OutOfHere•1d ago

Requiring PoW (proof-of-work) could take over for simple requests, rejecting requests until a sufficient nonce is included in the request. Unfortunately, this collective PoW could burden power grids even more, wasting energy+money+computation for transmission. Such is life. It would be a lot better to just upgrade the servers, but that's never going to be sufficient.

Bjartr•1d ago

So, Anubis?

https://anubis.techaro.lol/

OutOfHere•1d ago

Yes, although the concept is simple enough in principle that a homegrown solution also works.

Zardoz84•1d ago

We are wasting power on feeding statistics parrots, and we need to waste additional power to avoid being DoS by that feeding.

We will be better without that useless waste of power.

treyd•1d ago

What do you suppose we as website owners do to prevent our websites from being DoSed in the meantime? And how do you suppose we convince/beg the corporations running AI scraping bots to be better users of the web?

OutOfHere•1d ago

This should be an easy question for an engineer. It depends on whether the constraint is CPU or memory or database or network.

zihotki•23h ago

Technology can't solve a human problem, the constraints are in budgets and in available time

OutOfHere•5h ago

What human problem. Do tell -- how have sites handled search engine crawlers for the past few decades? Why are AI crawlers functionally different? It makes no sense because they aren't functionally different.

OutOfHere•3h ago

As of this year, AI has given people superpowers, doubling what they can achieve without it. Is this gain not enough? One can use it to run a more efficient web server.

jaoane•23h ago

Write proper websites that do not choke that easily.

HumanOstrich•17h ago

So I just need a solution with infinite compute, storage, and bandwidth. Got it.

jaoane•16h ago

That is not what I said and that is not what is necessary.

First of all web developers should use google and learn what a cache is. That way you don’t need compute at all.

throwawayscrapd•15h ago

And maybe you could Bing and learn what "cache eviction" is and why that happens when a crawler systematically hits every page on your site.

OutOfHere•5h ago

Maybe because it's an overly simplistic LRU cache, in which case a different eviction algorithm would be better.

It's funny really since Google and other search engines have been crawling sites for decades, but now that search engines have competition, sites are complaining.

OutOfHere•5h ago

How did you manage search engine crawlers for the past few decades? And why are AI crawlers functionally different? They aren't.

atonse•1d ago

How was this not a problem before with search engine crawlers?

Is this more of an issue with having 500 crawlers rather than any single one behaving badly?

Ndymium•1d ago

Search engine crawlers generally respected robots.txt and limited themselves to a trickle of requests, likely based on the relative popularity of the website. These bots do neither, they will crawl anything they can access and send enough requests per second to drown your server, especially if you're a self hoster running your own little site on a dinky server.

Search engines never took my site down, these bots did.

atonse•14h ago

Thanks for specifying the actual issue. As someone who hosts a bunch of sites, we're also seeing a spike in traffic, but we don't track user agents.

OutOfHere•5h ago

Maybe stop using an inefficient PHP/Javascript/Typescript server, and start using a more efficient Go/Rust/Nim/Zig server.

fogx•23h ago

esp. for image data libraries, why not provide the images as a dump instead? No need to crawl 3mil images if the download button is right there. Now put the file on a cdn or Google and you're golden

HumanOstrich•17h ago

There are two immediate issues I see with that. First, you'll end up with bots downloading the dump over and over again. Second, for non-trivial amounts of data, you'll end up paying the CDN for bandwidth anyway.

throwawayscrapd•16h ago

I work on the kind of big online scientific database that this article is about.

100% of our data is available from a clearly marked "Download" page.

We still have scraper bots running through the whole site constantly.

We are not "golden".

Chrome MCP: Open-source plugin to let any chatbot control your Chrome

Michael Truell(CEO, Cursor) on betting everything on a world beyond code

Disney, Universal File First Major Studio Lawsuit Against AI Company Midjourney

Quantity Kills

Is the 'tech bro-ification' of abortion here?

The Highest Form of Culinary Reverence: Ikizukuri

GM's silent about-face from EV production after losing $6B

Text-to-LoRA: Hypernetwork that generates task-specific LLM adapters (LoRAs)

Text-to-LoRA: Hypernetwork that generates task-specific LLM adapters (LoRAs)

EU launches EU-based, privacy-focused DNS resolution service

Darknet Market Maximalism

Navy backs right to repair after $13B carrier goes half-fed

Before the far right threatened democracy, neoliberalism stripped it down

The Law of the Excluded Middle

A Comprehensive Look at In-Ear EEG Electrodes and Their Applications

Brian Wilson has passed away

Ilya Sutskever Speech: Honorary Doctor of Science from University of Toronto [video]

OsmAnd's Faster Offline Navigation

AOSP project is coming to an end

The Centralization of the Internet

Fulbright Board Resigns After Accusing Trump Aides of Political Interference

Expanding Racks [video]

Lysenkoism

Ubuntu 25.10 Replaces Sudo with a Rust-Based Equivalent

Continental divide: Smaller Western European cities are better for your health

9-figure packages for AI Researchers at Meta

Concept-Centric Software Development (2023)

In case of emergency, break glass

Ubuntu 25.10 drops support for GNOME on Xorg

Cathemerality

Chrome MCP: Open-source plugin to let any chatbot control your Chrome

Michael Truell(CEO, Cursor) on betting everything on a world beyond code

Disney, Universal File First Major Studio Lawsuit Against AI Company Midjourney

Quantity Kills

Is the 'tech bro-ification' of abortion here?

The Highest Form of Culinary Reverence: Ikizukuri

GM's silent about-face from EV production after losing $6B

Text-to-LoRA: Hypernetwork that generates task-specific LLM adapters (LoRAs)

Text-to-LoRA: Hypernetwork that generates task-specific LLM adapters (LoRAs)

EU launches EU-based, privacy-focused DNS resolution service

Darknet Market Maximalism

Navy backs right to repair after $13B carrier goes half-fed

Before the far right threatened democracy, neoliberalism stripped it down

The Law of the Excluded Middle

A Comprehensive Look at In-Ear EEG Electrodes and Their Applications

Brian Wilson has passed away

Ilya Sutskever Speech: Honorary Doctor of Science from University of Toronto [video]

OsmAnd's Faster Offline Navigation

AOSP project is coming to an end

The Centralization of the Internet

Fulbright Board Resigns After Accusing Trump Aides of Political Interference

Expanding Racks [video]

Lysenkoism

Ubuntu 25.10 Replaces Sudo with a Rust-Based Equivalent

Continental divide: Smaller Western European cities are better for your health

9-figure packages for AI Researchers at Meta

Concept-Centric Software Development (2023)

In case of emergency, break glass

Ubuntu 25.10 drops support for GNOME on Xorg

Cathemerality

Web-scraping AI bots cause disruption for scientific databases and journals

Comments