frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

Ask HN: Who is running the abusive scrapers?

9•jonatron•4h ago
Personally I once had to triple app capacity and double database capacity just for just one scraper. I remember Cloudflare once figured out that Perplexity was one of the bad guys ( https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/ ) but has anyone else figured out who else is running large scale and poorly behaved scrapers?

Comments

mmakeev•4h ago
never got a name, best we managed was network attribution. ours came mostly from tencent cloud plus a second hetzner asn, all spoofed chrome on windows user agents, and it rotated from digitalocean to tencent within a day once we started blocking asns. need to tell that it executed the AWS WAF js challenge and replayed the token, but never solved an interactive captcha, so flipping the action from challenge to captcha is what finally stopped it
Bender•4h ago
Who is running the abusive scrapers?

Don't know, I just block them. These [1] are some of the methods I use with exception to method 7. I need to make time to try out the SSL fingerprint detection. They are crude methods but most of the bots / scrapers are kicked to the curb. The feds shutting down NetNut helped a lot too. I am looking forward to them shutting down more residential proxies.

Mirror [2]

[1] - https://nochan.net/b/Internet-Crap/20260606-How-To-Block-Som...

[2] - https://archive.is/kVLJZ

Ask HN: I am not able to find a job. Should I switch stacks?

7•need_a_work23•2h ago•7 comments

How to feel safe delegating to AI agents?

2•danicuki•2h ago•0 comments

Ask HN: Any good replacements for "Refined Hacker News?"

2•gwbas1c•3h ago•1 comments

Ask HN: Is Google NERFing AI Mode?

3•PaulHoule•3h ago•3 comments

Ask HN: What's the Fuzz about Taste?

2•pls-dnt-deploy•3h ago•2 comments

Ask HN: Who is running the abusive scrapers?

9•jonatron•4h ago•2 comments

Ask HN: HN frontpage feels boring now?

40•xeonmc•17h ago•27 comments

Ask HN: I hate coding agents. Is this skill issue?

4•cmar00•5h ago•4 comments

Ask HN: How long has it been since you last opened Stack Overflow?

3•novoreorx•5h ago•6 comments

Asker: A Socratic AI auditing agent using 3D cross-verification

2•tygerlee•6h ago•0 comments

Ask HN: What would you do if you had coded a better web browser than Firefox?

4•roschdal•2h ago•3 comments

Dev Jr Developer

3•edoschenone•7h ago•1 comments

CashFlow – Open-source debt minimization using graph algorithms and WASM

2•SanishKumar•8h ago•0 comments

Ask HN: Where are the good search engines for mathematical formulas?

56•lo0dot0•4d ago•16 comments

Teller.io shuts down at the end of the week

4•dmonn•10h ago•0 comments

Ask HN: Is GitHub preparing to go behind a login wall?

55•reconnecting•1d ago•38 comments

Ask HN: Is anyone experimenting with different ways of using LLMs for coding?

204•yehiaabdelm•6d ago•203 comments

Ask HN: How you manage local long lived research projects and LLM's?

6•ripplefringe•21h ago•2 comments

non-tech founder + choosing between buying ClaudeCode or CodeX

2•virtuallymundo•13h ago•5 comments

Ask HN: How to recover from thin content or de-indexing penalty on Google

4•anitroves•13h ago•4 comments

Tell HN: Who wants to be hired" posts outpace "Who's hiring" 2 to 1

55•santiagobasulto•2d ago•34 comments

Ask HN: Would you pay to discuss a book with an official AI clone of its author?

4•briangao•14h ago•13 comments

Ask HN: Are systems ready for the first negative leap second?

11•Asmod4n•3d ago•11 comments

Ask HN: Who is hiring? (July 2026)

246•whoishiring•1w ago•396 comments

Ask HN: Best Podcasts of 2026 [So Far]

20•AbstractH24•1d ago•19 comments

VentureBeat Events in San Francisco?

2•BriStoller•20h ago•0 comments

Ask HN: Opinionated Agent Setup?

3•lookACamel•21h ago•2 comments

Tell HN: I managed to unsubscribe from Adobe CC without being charged

26•frereubu•3d ago•3 comments

Ask HN: What are agent sandboxes missing?

5•oryx1729•22h ago•3 comments

Nginx Vulnerabilities

2•finbot•23h ago•4 comments