frontpage.
newsnewestaskshowjobs

Made with ♥ by @iamnishanth

Open Source @Github

fp.

Show HN: Mermaid Formatter – CLI and library to auto-format Mermaid diagrams

https://github.com/chenyanchen/mermaid-formatter
1•astm•9m ago•0 comments

RFCs vs. READMEs: The Evolution of Protocols

https://h3manth.com/scribe/rfcs-vs-readmes/
1•init0•15m ago•1 comments

Kanchipuram Saris and Thinking Machines

https://altermag.com/articles/kanchipuram-saris-and-thinking-machines
1•trojanalert•15m ago•0 comments

Chinese chemical supplier causes global baby formula recall

https://www.reuters.com/business/healthcare-pharmaceuticals/nestle-widens-french-infant-formula-r...
1•fkdk•18m ago•0 comments

I've used AI to write 100% of my code for a year as an engineer

https://old.reddit.com/r/ClaudeCode/comments/1qxvobt/ive_used_ai_to_write_100_of_my_code_for_1_ye...
1•ukuina•20m ago•1 comments

Looking for 4 Autistic Co-Founders for AI Startup (Equity-Based)

1•au-ai-aisl•31m ago•1 comments

AI-native capabilities, a new API Catalog, and updated plans and pricing

https://blog.postman.com/new-capabilities-march-2026/
1•thunderbong•31m ago•0 comments

What changed in tech from 2010 to 2020?

https://www.tedsanders.com/what-changed-in-tech-from-2010-to-2020/
2•endorphine•36m ago•0 comments

From Human Ergonomics to Agent Ergonomics

https://wesmckinney.com/blog/agent-ergonomics/
1•Anon84•40m ago•0 comments

Advanced Inertial Reference Sphere

https://en.wikipedia.org/wiki/Advanced_Inertial_Reference_Sphere
1•cyanf•41m ago•0 comments

Toyota Developing a Console-Grade, Open-Source Game Engine with Flutter and Dart

https://www.phoronix.com/news/Fluorite-Toyota-Game-Engine
1•computer23•43m ago•0 comments

Typing for Love or Money: The Hidden Labor Behind Modern Literary Masterpieces

https://publicdomainreview.org/essay/typing-for-love-or-money/
1•prismatic•44m ago•0 comments

Show HN: A longitudinal health record built from fragmented medical data

https://myaether.live
1•takmak007•47m ago•0 comments

CoreWeave's $30B Bet on GPU Market Infrastructure

https://davefriedman.substack.com/p/coreweaves-30-billion-bet-on-gpu
1•gmays•58m ago•0 comments

Creating and Hosting a Static Website on Cloudflare for Free

https://benjaminsmallwood.com/blog/creating-and-hosting-a-static-website-on-cloudflare-for-free/
1•bensmallwood•1h ago•1 comments

"The Stanford scam proves America is becoming a nation of grifters"

https://www.thetimes.com/us/news-today/article/students-stanford-grifters-ivy-league-w2g5z768z
3•cwwc•1h ago•0 comments

Elon Musk on Space GPUs, AI, Optimus, and His Manufacturing Method

https://cheekypint.substack.com/p/elon-musk-on-space-gpus-ai-optimus
2•simonebrunozzi•1h ago•0 comments

X (Twitter) is back with a new X API Pay-Per-Use model

https://developer.x.com/
3•eeko_systems•1h ago•0 comments

Zlob.h 100% POSIX and glibc compatible globbing lib that is faste and better

https://github.com/dmtrKovalenko/zlob
3•neogoose•1h ago•1 comments

Show HN: Deterministic signal triangulation using a fixed .72% variance constant

https://github.com/mabrucker85-prog/Project_Lance_Core
2•mav5431•1h ago•1 comments

Scientists Discover Levitating Time Crystals You Can Hold, Defy Newton’s 3rd Law

https://phys.org/news/2026-02-scientists-levitating-crystals.html
3•sizzle•1h ago•0 comments

When Michelangelo Met Titian

https://www.wsj.com/arts-culture/books/michelangelo-titian-review-the-renaissances-odd-couple-e34...
1•keiferski•1h ago•0 comments

Solving NYT Pips with DLX

https://github.com/DonoG/NYTPips4Processing
1•impossiblecode•1h ago•1 comments

Baldur's Gate to be turned into TV series – without the game's developers

https://www.bbc.com/news/articles/c24g457y534o
3•vunderba•1h ago•0 comments

Interview with 'Just use a VPS' bro (OpenClaw version) [video]

https://www.youtube.com/watch?v=40SnEd1RWUU
2•dangtony98•1h ago•0 comments

EchoJEPA: Latent Predictive Foundation Model for Echocardiography

https://github.com/bowang-lab/EchoJEPA
1•euvin•1h ago•0 comments

Disablling Go Telemetry

https://go.dev/doc/telemetry
1•1vuio0pswjnm7•1h ago•0 comments

Effective Nihilism

https://www.effectivenihilism.org/
1•abetusk•1h ago•1 comments

The UK government didn't want you to see this report on ecosystem collapse

https://www.theguardian.com/commentisfree/2026/jan/27/uk-government-report-ecosystem-collapse-foi...
5•pabs3•1h ago•0 comments

No 10 blocks report on impact of rainforest collapse on food prices

https://www.thetimes.com/uk/environment/article/no-10-blocks-report-on-impact-of-rainforest-colla...
3•pabs3•1h ago•0 comments
Open in hackernews

OpenStreetMap overwhelmed by bots scraping data

https://twitter.com/openstreetmap/status/2016320492420878531
45•molly_radstowe•1w ago

Comments

molly_radstowe•1w ago
#OpenStreetMap hammered by scrapers hiding behind residential proxy/embedded-SDK networks.
direwolf20•1w ago
More like hammered by Google and Apple so you'll use their apps instead.
petre•1w ago
Unlikely. The data is freely available for download from geofabrik and other sources.
direwolf20•1w ago
The data is, the app isn't. OSM provides a giant data dump, not a way to view maps
Bender•1w ago
Looks like it is hosted in Equinix in NL? Or just part of it maybe? Is it behind a load balancer, maybe something like HAProxy? If so were stick tables set up to limit rates by cookie and require people be logged in on unique accounts and limit anonymous access after so many requests? I know limiting anonymous access is not great but that is something that could be enabled when under a high load so that instead of the site going offline for everyone it would just be limited for the anonymous users. Degradation vs critical outage

On a separate note have tcpdump captures been done on these excessive connections? Minus the IP, what do their SYN packets look like? Minus the IP what do the corresponding log entries look like in the web server? Are they using HTTP/1.1 or HTTP/2.0? Are they missing any expected headers for a real person such as cors, no-cors, navigate, accept_language?

    tcpdump -p --dont-verify-checksums -i any -NNnnvvv -B32768 -c32 -s0 port 443 and 'tcp[13] == 2'
Is there someone at OpenStreetMap that can answer these questions?
KomoD•1w ago
I think it could be worth trying to block them with TLS fingerprinting, or since they think it's residential proxies they are being hammered by, https://spur.us could be worth a try.
Bender•1w ago
My personal preference is to first make a small amount of effort finding something unique to the bots that can more often than not be dropped with a simple firewall rule or load balancer ACL. The botters almost always miss something.
Firefishy•1w ago
Disclosure: I am part of the mostly volunteer run OpenStreetMap ops team.

Technically we able to block and restrict the scrapers after the initial request from an IP. We've seen 400,000 IPs in the last 24 hours. Each IP only does a few requests. Most are not very good at faking browsers, but they are getting better. (HTTP/1.1 vs HTTP/2, obviously faked headers etc)

The problem has been going on for over a year now. It isn't going away. We need journalists and others to help us push back.

Bender•1w ago
I hear ya. This is just my opinion but I don't think journalists are going to be much help. The bots would have to be hurting something belonging to the government or the government is paying for to really get them on it. e.g. some big orgs in the government embed your maps on their site. They would have to create legislation and then someone would have to trace the bots back to their operator for attribution and then someone would have to file lawsuits against them once it is illegal. Or you could try using a ToS/AuP to go after them assuming attribution. I am not a lawyer.

I think your only hope would be to either find subtle differences between them and real legit users or change how your site works so that bots have to be authenticated unless they have a whitelisted IP/CIDR or put your site behind something else that spots the bots. Beyond that all anyone can do is beef up their infrastructure to handle much more than the bots could dish out.

Have you tried silly simple things like hidden javascript puzzles the browser has to solve?

Kodiack•4d ago
Hey. I run a small community forum and I've been dealing with this exact same kind of behaviour where well over 99% of requests are bad crawlers. There used to be plenty of "tells" for the faked browsers, HTTP/1.1 being a huge one. As you said, however, they're getting a bit smarter about that and it's becoming increasingly difficult to differentiate it from legitimate traffic.

It's been getting worse over the past year, with the past few weeks in particular seeing a massive change literally overnight. I had to aggressively tune my WAF rules to even remotely get things under control. With Cloudflare I'm aggressively issuing browser challenges to any browser that looks remotely suspicious, and the pass rate is currently below 0.5%. For my users' sake, a successful browser challenge is "valid" for over a month, but this still feels like another thing that'll eventually be bypassed.

I'd be keen to know if you've found any other effective ways of mitigating these most recent aggressive scraping requests. Even a simple "yes" or "no" would be appreciated; I think it's fair to be apprehensive about sharing some specific details publicly since even a lot of folks here on HN seem to think it's their right to scrape content with orders of magnitude higher throughput than all users combined.

I really don't know how this is sustainable long-term. It's eaten up quite a lot of my personal time and effort just for the sake of a hobby that I otherwise greatly enjoy.

phillipseamore•1w ago
The number of idiotic vibe coded repos I've seen on GH lately that are doing things like crawling OSM for POI data is mindboggling!
CqtGLRGcukpy•1w ago
https://xcancel.com/openstreetmap/status/2016320492420878531

https://nitter.poast.org/openstreetmap/status/20163204924208...

CqtGLRGcukpy•1w ago
They also posted about this on Mastodon / Fedi: https://en.osm.town/@osm_tech/115968544599864782
dzhiurgis•1w ago
I'll ask dumb question - if they are "open source" then why they are bothered by it? Is it scraping itself? Are their data not freely available for download?
wodenokoto•1w ago
Someone has to pay for bandwidth. And that someone would like the bandwidth to go to human users.
zeeZ•1w ago
Their data is freely available to download. There are weekly dumps of the entire planet and several sources for partial data. There's no need for most legitimate use cases to scrape their API.
dzhiurgis•1w ago
So problem is someone is stupid enough scraping without realizing they can just download 100gb at once?

And there are so many of such idiots that it's overwhelming their servers?

Something doesn't math here.

tencentshill•1w ago
That's exactly the case. It's been an issue for a while.
solaris2007•1w ago
Make the data available through bit-torrent and IPFS. Redirect IPs that make excessive requests to response only kilobytes in size "use the torrents and IPFS".

As an SRE, the only legitimate concern here could be the bandwidth costs. But QoS tuning should solve that too.

Supposedly technical people crying out for a journalist to help them is super lame. Everything about this looks super lame.

zeeZ•1w ago
That data is already available. Including torrents.

https://planet.openstreetmap.org/

solaris2007•1w ago
Perfect. Now all they need to do is set up the redirect.

Every bot is doing something on behalf of a human. Now that LLMs can churn out half-assed bot scripts every "look I installed Arch Linux and ohmyzsh" script kiddie has bots too.

Bots aren't going anywhere.

"Use the web the way it was over 10 years ago plox" isn't going to do it.

Firefishy•1w ago
Disclosure: I am part of the OpenStreetMap mostly-volunteer sysadmin team fighting this.

The scrapers try hard to make themselves look like valid browsers, sending requests via residential IP addresses (400,000+ IPs at last count).

I reached out to journalists because despite strong technical measures, the abuse will not go away on its own.