frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

An Update on the scraper situation

https://lwn.net/SubscriberLink/1080822/990a8a5e2d379085/
22•chmaynard•1h ago

Comments

cyanydeez•1h ago
mmm, in many cases these residential proxies are media boxes, and they consent as much as anyone else consents to what amazon, or google or facebook does; it's buried somewhere in the recesses of the TOS.

The question is more about why the US and others can't properly enforce the bullshit all this amounts to.

tingletech•51m ago
The comments are not showing up for me now, but when they were still showing for anonymous users, there was a link to https://commoncrawl.org. I've been sort of worried about letting agents hit websites, I wonder if a fetch_url agent tool could be made to look in common crawl first before hitting the web for it?
atomic128•49m ago
There is a large community of sites feeding poison to scrapers.

The poison gets better every day, and the community gets larger.

For example, here is one part of the community on Reddit: https://www.reddit.com/r/PoisonFountain/comments/1uocaii/a_n...

logancbrown•11m ago
People think this is causing issues for data collection for LLMs, but in reality it's not and there are several very trivial mechanisms to employ in data collection to bypass the "poison data" issue. The internet landscape was already poisoned with fake data, fringe conspiracies, and text before this Poison Fountain initiative.
mips_avatar•49m ago
I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs.

I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.

everfrustrated•21m ago
I wonder how much of this is traffic caused by peoples agents using web tools causing searches and fetches rather than general trawls of the internet.
corbet•3m ago
Very little of it. When you see a million IPs systematically working their way through your URL space, it's pretty clear that there's a central control node behind it all.
sixtyj•10m ago
The issue with scrapping is the intensity and volume of bots.

I think that nobody would care if I use wget or curl for few pages, e.g. because I would like to read a site as offline or archive it.

Btw average age of any page is 10 years. Deletion or structural change after acquisition is common, Signal vs Noise site recent wipe out could serve as an example why we need to archive sites.

QuadRF can spot drones and see WiFi through my wall

https://www.jeffgeerling.com/blog/2026/quadrf-can-spot-drones-and-see-wifi-through-my-wall/
346•speckx•5h ago•134 comments

GPT-5.6, Grok 4.5, Claude, and Muse Spark build the same 4 apps

https://www.tryai.dev/blog/gpt-5.6-build-off-12-models
14•hershyb_•29m ago•2 comments

Snails' teeth beats spider silk as nature's strongest material (2015)

https://www.smithsonianmag.com/smart-news/spider-silk-loses-top-spot-natures-strongest-material-s...
127•simonebrunozzi•4h ago•86 comments

GPT-5.6 Sol Ultra produces proof of the Cycle Double Cover Conjecture [pdf]

https://cdn.openai.com/pdf/04d1d1e4-bc75-476a-97cf-49055cd98d31/cdc_proof.pdf
203•scrlk•2h ago•181 comments

The tech of 'Terminator 2' – an oral history (2017)

https://vfxblog.com/2017/08/23/the-tech-of-terminator-2-an-oral-history/
116•markus_zhang•4h ago•47 comments

Moss (YC F25) Is Hiring

https://www.ycombinator.com/companies/moss/jobs/52LnqLQ-software-engineer-sdk
1•srimalireddi•10m ago

How the terrorist group Boko Haram uses frontier AI

https://casp.ac/reports/ai-enabled-terrorism
83•imustachyou•2h ago•68 comments

War Atlas: An interactive cartography of every named war in human history

https://waratlas.org
68•NaOH•3h ago•23 comments

New York City to become first in US to ban deceptive subscription practices

https://www.theguardian.com/us-news/2026/jul/10/new-york-city-deceptive-subscriptions-ban
236•randycupertino•2h ago•84 comments

Combustion Engine Web-Based Simulator

https://combustionlab.net
69•mytuny•5d ago•28 comments

Don't discontinue Gemini 2.5 Flash

https://discuss.ai.google.dev/t/please-dont-discontinue-gemini-2-5-flash/174246
26•NickDob•1h ago•14 comments

Computation as a universal and fundamental concept

https://ergo.org/courses/computation-as-a-universal-and-fundamental-concept
56•simonpure•5h ago•53 comments

An Update on the scraper situation

https://lwn.net/SubscriberLink/1080822/990a8a5e2d379085/
23•chmaynard•1h ago•9 comments

Show HN: Wyrm – Solve algebra by touch, built on an open-source soundness engine

https://github.com/dicroce/wyrm_math
25•dicroce•1d ago•2 comments

Late Bronze Age Collapse

https://acoup.blog/2026/01/30/collections-the-late-bronze-age-collapse-a-very-brief-introduction/
279•dmonay•9h ago•189 comments

Good Tools Are Invisible

https://www.gingerbill.org/article/2026/07/10/good-tools-are-invisible/
293•theanonymousone•10h ago•140 comments

Why We Don't Trust the Database with Authentication

https://blog.sturdystatistics.com/posts/api_keys/
9•kianN•3d ago•1 comments

Lost city discovered beneath Egypt's desert with ancient church

https://www.dailymail.com/sciencetech/article-15956159/Incredible-lost-city-discovered-Egypts-des...
122•Bender•4d ago•62 comments

Materials innovation has a scale-up problem, not discovery

https://www.atomscale.ai/updates/our-thesis-atom-to-scale
17•groznyj•2h ago•3 comments

The Clouds of Hiroshima

https://doomsdaymachines.net/p/the-clouds-of-hiroshima
15•handfuloflight•3d ago•11 comments

Successful Companies Go Blind

https://ianreppel.org/how-successful-companies-go-blind/
160•speckx•7h ago•57 comments

A Love Letter to Flashcards

https://lesleylai.info/en/flashcards/
108•surprisetalk•5h ago•65 comments

Write code like a human will maintain it

https://unstack.io/write-code-like-a-human-will-maintain-it
301•ScottWRobinson•7h ago•247 comments

Show HN: Reviving my 2001 college band with AI

https://www.fadingmaize.com
42•jacobgraf•1d ago•49 comments

Show HN: Reverse-engineering web apps into agent tools

74•pancomplex•1d ago•26 comments

45% of Enthusiasts 'Seriously Considering' Leaving Sony for PC

https://www.pushsquare.com/news/2026/07/ps5-has-put-a-dampener-on-gaming-45percent-of-enthusiasts...
54•speckx•2h ago•70 comments

The mathematical secrets of Barcelona's Sagrada Familia

https://mappingignorance.org/2026/06/30/sagrada-familia/
111•Gedxx•1w ago•27 comments

Cpp2Rust: Translates C++ to safe Rust automatically

https://github.com/Cpp2Rust/cpp2rust
32•signa11•4h ago•3 comments

Hands-On with the AMD Ryzen AI Halo

https://www.microcenter.com/site/mc-news/article/amd-ryzen-ai-halo-review.aspx
35•bdcravens•5h ago•31 comments

Ask HN: Are systems ready for the first negative leap second?

40•Asmod4n•4d ago•45 comments