frontpage.
newsnewestaskshowjobs

Made with ♥ by @iamnishanth

Open Source @Github

fp.

Anthropic: Latest Claude model finds more than 500 vulnerabilities

https://www.scworld.com/news/anthropic-latest-claude-model-finds-more-than-500-vulnerabilities
1•Bender•3m ago•0 comments

Brooklyn cemetery plans human composting option, stirring interest and debate

https://www.cbsnews.com/newyork/news/brooklyn-green-wood-cemetery-human-composting/
1•geox•3m ago•0 comments

Why the 'Strivers' Are Right

https://greyenlightenment.com/2026/02/03/the-strivers-were-right-all-along/
1•paulpauper•4m ago•0 comments

Brain Dumps as a Literary Form

https://davegriffith.substack.com/p/brain-dumps-as-a-literary-form
1•gmays•5m ago•0 comments

Agentic Coding and the Problem of Oracles

https://epkconsulting.substack.com/p/agentic-coding-and-the-problem-of
1•qingsworkshop•5m ago•0 comments

Malicious packages for dYdX cryptocurrency exchange empties user wallets

https://arstechnica.com/security/2026/02/malicious-packages-for-dydx-cryptocurrency-exchange-empt...
1•Bender•5m ago•0 comments

Show HN: I built a <400ms latency voice agent that runs on a 4gb vram GTX 1650"

https://github.com/pheonix-delta/axiom-voice-agent
1•shubham-coder•6m ago•0 comments

Penisgate erupts at Olympics; scandal exposes risks of bulking your bulge

https://arstechnica.com/health/2026/02/penisgate-erupts-at-olympics-scandal-exposes-risks-of-bulk...
3•Bender•7m ago•0 comments

Arcan Explained: A browser for different webs

https://arcan-fe.com/2026/01/26/arcan-explained-a-browser-for-different-webs/
1•fanf2•8m ago•0 comments

What did we learn from the AI Village in 2025?

https://theaidigest.org/village/blog/what-we-learned-2025
1•mrkO99•9m ago•0 comments

An open replacement for the IBM 3174 Establishment Controller

https://github.com/lowobservable/oec
1•bri3d•11m ago•0 comments

The P in PGP isn't for pain: encrypting emails in the browser

https://ckardaris.github.io/blog/2026/02/07/encrypted-email.html
2•ckardaris•13m ago•0 comments

Show HN: Mirror Parliament where users vote on top of politicians and draft laws

https://github.com/fokdelafons/lustra
1•fokdelafons•14m ago•1 comments

Ask HN: Opus 4.6 ignoring instructions, how to use 4.5 in Claude Code instead?

1•Chance-Device•15m ago•0 comments

We Mourn Our Craft

https://nolanlawson.com/2026/02/07/we-mourn-our-craft/
1•ColinWright•18m ago•0 comments

Jim Fan calls pixels the ultimate motor controller

https://robotsandstartups.substack.com/p/humanoids-platform-urdf-kitchen-nvidias
1•robotlaunch•22m ago•0 comments

Exploring a Modern SMTPE 2110 Broadcast Truck with My Dad

https://www.jeffgeerling.com/blog/2026/exploring-a-modern-smpte-2110-broadcast-truck-with-my-dad/
1•HotGarbage•22m ago•0 comments

AI UX Playground: Real-world examples of AI interaction design

https://www.aiuxplayground.com/
1•javiercr•22m ago•0 comments

The Field Guide to Design Futures

https://designfutures.guide/
1•andyjohnson0•23m ago•0 comments

The Other Leverage in Software and AI

https://tomtunguz.com/the-other-leverage-in-software-and-ai/
1•gmays•25m ago•0 comments

AUR malware scanner written in Rust

https://github.com/Sohimaster/traur
3•sohimaster•27m ago•1 comments

Free FFmpeg API [video]

https://www.youtube.com/watch?v=6RAuSVa4MLI
3•harshalone•27m ago•1 comments

Are AI agents ready for the workplace? A new benchmark raises doubts

https://techcrunch.com/2026/01/22/are-ai-agents-ready-for-the-workplace-a-new-benchmark-raises-do...
2•PaulHoule•32m ago•0 comments

Show HN: AI Watermark and Stego Scanner

https://ulrischa.github.io/AIWatermarkDetector/
1•ulrischa•33m ago•0 comments

Clarity vs. complexity: the invisible work of subtraction

https://www.alexscamp.com/p/clarity-vs-complexity-the-invisible
1•dovhyi•34m ago•0 comments

Solid-State Freezer Needs No Refrigerants

https://spectrum.ieee.org/subzero-elastocaloric-cooling
2•Brajeshwar•34m ago•0 comments

Ask HN: Will LLMs/AI Decrease Human Intelligence and Make Expertise a Commodity?

1•mc-0•35m ago•1 comments

From Zero to Hero: A Brief Introduction to Spring Boot

https://jcob-sikorski.github.io/me/writing/from-zero-to-hello-world-spring-boot
1•jcob_sikorski•35m ago•1 comments

NSA detected phone call between foreign intelligence and person close to Trump

https://www.theguardian.com/us-news/2026/feb/07/nsa-foreign-intelligence-trump-whistleblower
14•c420•36m ago•2 comments

How to Fake a Robotics Result

https://itcanthink.substack.com/p/how-to-fake-a-robotics-result
1•ai_critic•36m ago•0 comments
Open in hackernews

Ask HN: What measures are you taking to stop AI crawlers?

6•kjok•5mo ago
Curious to know what steps people here are taking to protect their sites, products, and APIs. What have you tried that actually works in practice?

Comments

JohnFen•5mo ago
I spent a lot of time trying to find a good solution to this problem and failed, so what I ended up doing was to give up and remove my sites from the public web entirely.

I'm eager for a good solution that will allow me to put them back, but I'm doubtful that's going to happen. In any case, I'm extremely interested in other people's replies here. Maybe there's a solution that I haven't been able to find!

mmarian•5mo ago
I'm curious, what made you decide to completely remove them from the public web?
JohnFen•5mo ago
I couldn't find a way to protect them from AI crawlers.
mmarian•5mo ago
Indeed ^_^, but what I meant to ask is why did the possibility of AI crawlers fetching your site trigger such a big response? Especially when there's services/tools like Cloudflare and Anubis that solve that.

Some people are worried about the volume, but at worst your site is down for a while. Unless you're on serverless, but in that case you would've had to be worried about hackers before AI crawlers existed.

If it's IP theft, if someone's motivated enough to get past Cloudflare/Anubis, you would've had to worry about them before AI crawlers existed too.

JohnFen•5mo ago
I very much don't want my stuff to be used in training genAI models, and if even a single crawler finds its way to being able to get my data then it's game over because there's no way to get it excluded after the fact.

> Especially when there's services/tools like Cloudflare and Anubis that solve that.

They don't solve it. They offer some help, but it's far from good enough.

The exact reason why I don't want my stuff included isn't really relevant to the topic, honestly ("I don't like it" is plenty sufficient reason, after all), but I will say there are multiple reasons. The main issue is that if it happens, there's no way to correct for that.

IP violation is one of them, but not the main one -- but it does illustrate the point: in the case of more traditional IP "theft", there is technically a way to resolve the problem after it happens. Not so with genAI crawlers.

mmarian•5mo ago
I see, would love to ask more questions but I get the impression you'd like to stop here. Thanks for the conversation!
JohnFen•5mo ago
I try not to get into the reasons I don't want my stuff to be ingested by genAI if that's not the topic, because it inevitably results in the conversation derailing as people argue with my reasons. But they aren't a big secret.

I don't want my stuff to get used to train genAI because I think that genAI is, on the whole, a negative for society. Not because of anything inherent in the technology itself, but because of the way genAI companies are employing and marketing it. On top of that, I resent that genAI companies have abused my trust by slurping everything up to train the models without asking or even so much as a "how do you do" (this is a different than an IP law argument).

I also think that genAI companies are increasingly abusive and manipulative and don't want to support them in any way. And the fact that they often argue about how dangerous the technology is and the great risk to humanity it presents, but at the same time are eager to develop and sell it, indicates an extreme level of ethical bankruptcy.

mmarian•5mo ago
I think it's reasonable. I also find it frustrating that you don't get credit for the content you create. It's stopped me from writing more about virtual offices, for example.

And if you really want to prevent any chance of that happening, the only way is to take it down unfortunately.

chistev•5mo ago
what do you mean by removing from the public web?
JohnFen•5mo ago
I took a couple down entirely and put the others behind a login wall.
bediger4000•5mo ago
I have a lot of them in robots.txt as disallow /, of course. I have several getting 404 on any request whatsoever, Meta's AI crawler, Bytespider mainly, via Apache httpd mod_rewrite.
johng•5mo ago
Some of our sites have been getting absolutely hammered by the AI bots -- so much so they are taking down the sites. Even with cloudflare protection and caching. The only thing We've been able to do so far is tell Cloudflare to block all AI bots, modify the robots.txt and even then we've had to manually identify IP addresses and bots that ignore all of the above and block them specifically or at the ASN level.

Cloudflare makes doing this kind of stuff easy but I would hate to have to do this manually on a webserver. And I don't like the idea of how much of the internet already relies on Cloudflare.

ATechGuy•5mo ago
Just saw this https://x.com/ycombinator/status/1960779353589211577

They say "... can scrape any website—not even Cloudflare can detect it."

mmarian•5mo ago
I set up the Cloudflare blocks on one site where I don't want the content to be ingested. Seems to work pretty well, my SEO looks to be ok too.