frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

Ask HN: Who is running the abusive scrapers?

7•jonatron•1h ago
Personally I once had to triple app capacity and double database capacity just for just one scraper. I remember Cloudflare once figured out that Perplexity was one of the bad guys ( https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/ ) but has anyone else figured out who else is running large scale and poorly behaved scrapers?

Comments

mmakeev•32m ago
never got a name, best we managed was network attribution. ours came mostly from tencent cloud plus a second hetzner asn, all spoofed chrome on windows user agents, and it rotated from digitalocean to tencent within a day once we started blocking asns. need to tell that it executed the AWS WAF js challenge and replayed the token, but never solved an interactive captcha, so flipping the action from challenge to captcha is what finally stopped it
Bender•27m ago
Who is running the abusive scrapers?

Don't know, I just block them. These [1] are some of the methods I use with exception to method 7. I need to make time to try out the SSL fingerprint detection. They are crude methods but most of the bots / scrapers are kicked to the curb. The feds shutting down NetNut helped a lot too. I am looking forward to them shutting down more residential proxies.

Mirror [2]

[1] - https://nochan.net/b/Internet-Crap/20260606-How-To-Block-Som...

[2] - https://archive.is/kVLJZ

Building the AI Retrieval Infrastructure Behind 20B+ Vectors at HubSpot

https://product.hubspot.com/blog/building-the-ai-retrieval-infrastructure-behind-20-billion-vecto...
1•cyndunlop•2m ago•0 comments

Show HN: Kiyeovo 1.0 release- dual network-mode decentralized E2EE P2P messenger

https://github.com/Realman78/Kiyeovo/releases/tag/kiyeovo-1.0.0
4•Realman78•3m ago•0 comments

Files over tools: how we built our agent with a virtual filesystem and bash

https://knock.app/blog/how-we-built-the-knock-agent-virtual-filesystem-and-bash
3•cjbell•3m ago•0 comments

Interior Department asks: Can oil platforms be used for space launches?

https://www.washingtonpost.com/climate-environment/2026/07/09/interior-department-asks-can-oil-pl...
1•reaperducer•3m ago•0 comments

Quebec is the only province where you can pre-approve your death

https://www.cbc.ca/lite/story/9.7261821
1•colinprince•4m ago•0 comments

Show HN: Virena, a minimal vision-language-action robot model you can read

https://github.com/BuceaGeorgia/VIRENA
1•georgia_bucea•5m ago•0 comments

Riot Engineering: How VALORANT Solved Peeker's Advantage (2020)

https://www.riotgames.com/en/news/peeking-valorants-netcode
1•xeonmc•5m ago•0 comments

You're already living in the Chinese Century

https://www.wired.com/china-issue/
2•JSR_FDED•5m ago•0 comments

Show HN: I built a budgeting app where the AI does the bookkeeping

https://play.google.com/store/apps/details?id=com.wladradchenko.fehu&hl=en_US
1•wladradchenko•6m ago•0 comments

AI builders outnumber AI governance hires 7:1 in Europe

https://axipro.co/eu-ai-act-hiring-gap-study/
1•pmoorcraft•7m ago•0 comments

Ask HN: What's the Fuzz about Taste?

1•pls-dnt-deploy•7m ago•0 comments

The Seed Beneath the Snow

https://eli.li/the-seed-beneath-the-snow
1•surprisetalk•7m ago•0 comments

Floorp Browser

https://floorp.app/
2•thunderbong•11m ago•1 comments

The Hell of Tetra Master (2023, 2025)

https://xvw.lol/en/articles/tetra-master.html
1•wise_blood•11m ago•0 comments

LLMs for technical editing: The good, the bad, and the ugly

https://techstackups.com/articles/llms-for-technical-editing-the-good-the-bad-and-the-ugly/
1•ritzaco•12m ago•0 comments

Just Pay the Subscription

https://www.uncommonapps.nyc/p/just-pay-the-subscription
1•dabluck•12m ago•1 comments

Show HN: Slopera, a browser that hallucinates every page with an LLM

https://github.com/fresswolf/Slopera
1•fresswolf•12m ago•0 comments

India Is Building Surveillance Infrastructure on Broken Data and Bad Policing

https://www.techpolicy.press/india-is-building-surveillance-infrastructure-on-broken-data-and-bad...
3•cdrnsf•12m ago•0 comments

Beni Is a Cute Autonomous Camera Robot on Wheels

https://petapixel.com/2026/07/08/beni-is-a-cute-autonomous-camera-robot-on-wheels/
2•dxs•12m ago•0 comments

Command Line Interface Guidelines

https://clig.dev/
1•sodic•13m ago•0 comments

SpaceXAI launches Grok 4.5 model for coding, agentic tasks

https://www.reuters.com/business/media-telecom/spacexai-launches-grok-45-model-coding-agentic-tas...
2•adithyaharish•13m ago•0 comments

Show HN: Noema – AI personas debate US listed stocks

https://itsnoema.com
1•hobbescotch•13m ago•0 comments

Tracking unique visitors without cookies

https://inmargin.io/blog/tracking-unique-visitors-without-cookies
1•jamie_davenport•13m ago•0 comments

Show HN: OpenTab – a lazygit-style TUI for your AI token spend

https://github.com/hamidi-dev/opentab
1•hamidi-dev•15m ago•0 comments

A Road to Lisp: Why Lisp

https://scotto.me/blog/2026-07-09-why-lisp/
1•silcoon•16m ago•0 comments

Show HN: Yogen – 500 AI agents argue about your idea before you bet on it

https://github.com/michaelwrites67-ctrl/yogen
1•michaelruocco•16m ago•0 comments

Data Centers Are Quietly Taking over Texas. The Pollution Could Be Catastrophic

https://www.wired.com/story/data-centers-taking-over-texas-pollution-could-be-catastrophic/
4•littlexsparkee•17m ago•1 comments

Show HN: Wizard, Self-extending autonomous AI agent in one Rust binary

https://github.com/teddytennant/wizard
1•Theodoretennant•19m ago•1 comments

Backblaze Drive Stats for Q1 2026

https://www.backblaze.com/blog/backblaze-drive-stats-for-q1-2026/
1•HieronymusBosch•21m ago•0 comments

Listen, you're probably not good at the violin

https://homosabiens.substack.com/p/listen-youre-probably-not-good-at
1•surprisetalk•21m ago•0 comments