Blocking LLM crawlers without JavaScript

https://www.owl.is/blogg/blocking-crawlers-without-javascript/

31•todsacerdoti•4h ago

Comments

superkuh•1h ago

I thought this was cool because it worked even in my old browser. So cool I went to add their RSS feed to my feed reader. But then my feed reader got blocked by the system. So now it doesn't seem so cool.

If the site author reads this: make an exception for https://www.owl.is/blogg/index.xml

This is a common mistake and the author is in good company. Science.org once blocked all of their hosted blogs' feeds for 3 months when they deployed a default cloudflare setup across all their sites.

SquareWheel•1h ago

That may work for blocking bad automated crawlers, but an agent acting on behalf of a user wouldn't follow robots.txt. They'd run the risk of hitting the bad URL when trying to understand the page.

klodolph•17m ago

That sounds like the desired outcome here. Your agent should respect robots.txt, OR it should be designed to not follow links.

daveoc64•1h ago

Seems pretty easy to cause problems for other people with this.

If you follow the link at the end of my comment, you'll be flagged as an LLM.

You could put this in an img tag on a forum or similar and cause mischief.

Don't follow the link below:

https://www.owl.is/stick-och-brinn/

If you do follow that link, you can just clear cookies for the site to be unblocked.

kazinator•37m ago

You do not have a meta refresh timer that will skip your entire comment and redirect to the good page in a fraction of a second too short for a person to react.

You also have not used <p hidden> to conceal the paragraph with the link from human eyes.

petesergeant•1h ago

I wish blockers would distinguish between crawlers that index, and agentic crawlers serving an active user's request. npm blocking Claude Code is irritating

specialp•49m ago

Agentic crawlers are worse. I run a primary source site and the ai "thinking" user agents will hit your site 1000+ times in a minute at any time of the day

klodolph•15m ago

I think of those two, agentic crawlers are worse.

behnamoh•1h ago

Any ideas on how to block LLMs from reading/analyzing a PDF? I don't want to submit a paper to journals only for them to use ChatGPT to review it...

(it has happened before)

Edit: I'm starting to get downvoted. Perhaps by the lazy-ass journal reviewrs?

jadbox•33m ago

Short answer is no. There are pdf black magic DRM tricks that could be used, but most PDF libraries used for AIs will decode it, making it mute. It's better just to add a note for the humans that "This PDF is meant to best enjoyed by humans" or something of that note.

cortesoft•28m ago

If someone can read it, they can put it through an LLM. There is no possible way to prevent that. Even with crazy DRM, you could take a picture of your screen and OCR it.

They are trying to block automated LLM scraping, which at least has some possibility of having some success.

zb3•4m ago

There's a way - inject garbage prompts, like in the content meant to be the example - humans might understand that this is in an "example" context, but LLMs are likely to fail as prompt injection is an unsolved problem.

Springtime•55m ago

I wonder what the venn diagram of end users who disable Javascript and also block cookies by default looks like. As the former is already something users have to do very deliberately so I feel the likelihood of the latter among such users is higher.

There's no cookies disabled error handling on the site, so the page just infinitely reloads in such cases (Cloudflare's check for comparison informs the user cookies are required—even if JS is also disabled).

DeepYogurt•30m ago

Has anyone done a talk/blog/whatever on how llm crawlers are different than classical crawlers? I'm not up on the difference.

klodolph•18m ago

The only real difference that LLM crawlers tend to not respect /robots.txt and some of them hammer sites with some pretty heavy traffic.

The trap in the article has a link. Bots are instructed not to follow the link. The link is normally invisible to humans. A client that visits the link is probably therefore a poorly behaved bot.

nektro•23m ago

nice post

AirPods libreated from Apple's ecosystem

IDEmacs: A Visual Studio Code clone for Emacs

Our investigation into the suspicious pressure on Archive.today

libwifi: an 802.11 frame parsing and generation library written in C

Blocking LLM crawlers without JavaScript

When did people favor composition over inheritance?

The inconceivable types of Rust: How to make self-borrows safe (2024)

Things that aren't doing the thing

AsciiMath

When UPS charged me a $684 tariff on $355 of vintage computer parts

Boa: A standard-conforming embeddable JavaScript engine written in Rust

Transgenerational Epigenetic Inheritance: the story of learned avoidance

Computing Across America (1983-1985)

EyesOff: How I built a screen contact detection model

Show HN: Unflip – a puzzle game about XOR patterns of squares

Archimedes – A Python toolkit for hardware engineering

Linux on the Fujitsu Lifebook U729

I made a better DOM morphing algorithm

JVM exceptions are weird: a decompiler perspective

Report: Tim Cook could step down as Apple CEO 'as soon as next year'

TCP, the workhorse of the internet

The computer poetry of J. M. Coetzee's early programming career (2017)

Weighting an average to minimize variance

AMD continues to chip away at Intel's x86 market share

Nevada Governor's office covered up Boring Co safety violations

Trellis AI (YC W24) Is Hiring: Streamline access to life-saving therapies

Show HN: High-Performance .NET Bindings for the Vello Sparse Strips CPU Renderer

Solving Project Euler: Problem 45

Messing with scraper bots

Feature Extraction with KNN

Blocking LLM crawlers without JavaScript

Comments

AirPods libreated from Apple's ecosystem

IDEmacs: A Visual Studio Code clone for Emacs

Our investigation into the suspicious pressure on Archive.today

libwifi: an 802.11 frame parsing and generation library written in C

Blocking LLM crawlers without JavaScript

When did people favor composition over inheritance?

The inconceivable types of Rust: How to make self-borrows safe (2024)

Things that aren't doing the thing

AsciiMath

When UPS charged me a $684 tariff on $355 of vintage computer parts

Boa: A standard-conforming embeddable JavaScript engine written in Rust

Transgenerational Epigenetic Inheritance: the story of learned avoidance

Computing Across America (1983-1985)

EyesOff: How I built a screen contact detection model

Show HN: Unflip – a puzzle game about XOR patterns of squares

Archimedes – A Python toolkit for hardware engineering

Linux on the Fujitsu Lifebook U729

I made a better DOM morphing algorithm

JVM exceptions are weird: a decompiler perspective

Report: Tim Cook could step down as Apple CEO 'as soon as next year'

TCP, the workhorse of the internet

The computer poetry of J. M. Coetzee's early programming career (2017)

Weighting an average to minimize variance

AMD continues to chip away at Intel's x86 market share

Nevada Governor's office covered up Boring Co safety violations

Trellis AI (YC W24) Is Hiring: Streamline access to life-saving therapies

Show HN: High-Performance .NET Bindings for the Vello Sparse Strips CPU Renderer

Solving Project Euler: Problem 45

Messing with scraper bots

Feature Extraction with KNN