frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

One Wikipedia page costs your AI agent 68,000 tokens

8•arhamislam5766•3h ago
i use claude code daily and measured what pages cost it while doing research. an average wikipedia article, for instance, is 68,240 tokens of raw html (tiktoken); nike's homepage is 353,000.

claude code's built-in webfetch handles the easy case well. it summarizes wikipedia to about 950 tokens and clears cloudflare on some sites like indeed and ticketmaster. but, and there's always a but, on js-rendered and some anti-bot pages it returns nothing.

quotes.toscrape.com/js gives "no quotes found"; nike.com gives a 403. your agent then dumps the raw html back into context and still fails. (note: i have also had cases where i read through the chat at the end and saw that it failed and just pulled from either training data or stale caches from other sources)

so i worked on building an open-source stealth browser (recompiled chromium) that runs as an mcp for claude code, cursor, and claude desktop. essentially all i have to change form my end is add the mcp, and it returns the cleaned up tokens while also beating detection: the js quotes come back in 285 tokens, nike in about 700 instead of a 403.

there is still stuff i am actively working on: there's no residential egress yet, and it won't beat kasada-style walls. it's for agents, qa, and research.

repo and the reproducible benchmark: https://github.com/tiliondev/fortress/tree/main/mcp i'm the author and here for feedback.

Comments

arthurcolle•3h ago
one of the early jina.ai products was/is reader api, and they trained ReaderLM for this purpose. definitely a good idea to check out existing implementations
m463•2h ago
> on js-rendered and some anti-bot pages it returns nothing

yeah, and my (human piloted) browser gets blocked by so many websites now. I routinely get "Sorry." when trying to log into hn.

sigh.

dlcarrier•1h ago
Hacker News is the one page that hasn't assumed I'm a bot, yet.

At this rate, we'll need to use bots to browse the web for us, because they're the only way to get through the anti-bot filters.

chonghaoju•1h ago
stripping to markdown with Jina Reader or Trafilatura before passing to the agent cuts that 68k down to ~3-5k for most Wikipedia pages, and handles the JS-rendered case too.

Ask HN: Are systems ready for the first negative leap second?

63•Asmod4n•4d ago•63 comments

Ask HN: How do you use Vim in the era of AI?

35•rstagi•14h ago•73 comments

One Wikipedia page costs your AI agent 68,000 tokens

8•arhamislam5766•3h ago•4 comments

Ask HN: What was the last task where only a frontier model could do it?

23•thedebuglife•10h ago•26 comments

Tell HN: I De-Googled Myself

28•degoogled•13h ago•17 comments

Ask HN: Burned Tokens scented candle ideas

4•Tade0•14h ago•13 comments

Ask HN: Is anyone experimenting with different ways of using LLMs for coding?

207•yehiaabdelm•1w ago•204 comments

Ask HN: HN frontpage feels boring now?

46•xeonmc•2d ago•35 comments

Ask HN: What data have the frontier AI companies not ingested yet?

2•lysace•7h ago•3 comments

Ask HN: I hate coding agents. Is this skill issue?

16•cmar00•1d ago•14 comments

Ask HN: Where are the good search engines for mathematical formulas?

56•lo0dot0•6d ago•16 comments

Ask HN: I am not able to find a job. Should I switch stacks?

11•need_a_work23•1d ago•17 comments

Ask HN: Is GitHub preparing to go behind a login wall?

56•reconnecting•3d ago•38 comments

Fable July 12th disclaimer disappears from Claude Code

7•gorgmah•1d ago•7 comments

Ask HN: Ubuntu 24.04 now force-automounts any plugged-in MMC card or USB key

3•everybodyknows•1d ago•1 comments

Ask HN: Who is hiring? (July 2026)

246•whoishiring•1w ago•403 comments

Tell HN: Who wants to be hired" posts outpace "Who's hiring" 2 to 1

55•santiagobasulto•3d ago•34 comments

Who has been caught out by this anti-pattern from Google?

4•Monotoko•1d ago•2 comments

Audar-ASR-V1:Arabic-first speech recognition foundation models with open weights

2•AudarAI•1d ago•2 comments

Ask HN: Thoughts on AI in Personal Finance?

6•foxtrot8672•1d ago•11 comments

Ask HN: Is Google NERFing AI Mode?

4•PaulHoule•1d ago•4 comments

Ask HN: What are you running locally on your machine with LLMs?

3•agcat•1d ago•3 comments

Ask HN: How long has it been since you last opened Stack Overflow?

5•novoreorx•1d ago•12 comments

Where'd Codex Go?

8•aleshh•1d ago•14 comments

You've reached the end!