Meta's Muse is fantastic for web scraping

https://sigh.dev/posts/metas-muse-is-fantastic-for-web-scraping/

25•STRiDEX•1h ago

Comments

dvt•37m ago

I built an AI "web harness" running on a sandboxed Chromium (using a custom side-loaded plugin that talks over websockets to a "driver") to basically do anything a normal user could do in a browser. It totally bypasses any and all bot measures and only gets the ones you yourself would get as well (and passes those successfully, e.g. Cloudflare checkbox or those annoying OCR puzzles).

Not sure if I should release it, but I'm sure more people are catching onto the power of agentic browsing.

ares623•31m ago

By describing it here you've already released it no?

d0vs•28m ago

FYI Muse smashes through those captchas natively without prompting.

frabcus•27m ago

Right, but what happens when everyone uses that at scale? Without agreements and standards it isn't pretty.

dvt•15m ago

It's sad, but this train has left the station a quarter of a century ago imo. Many people have since become billionaires scraping the web without anyone agreeing (Google, Yahoo, and now possibly Anthropic and OpenAI).

bayindirh•26m ago

Thanks for letting us know that we need a new layer of detection systems.

Also it’s great(!) to see that we’re going from “but ethics” to “I got mine, who cares”.

Humans are interesting creatures.

pcthrowaway•11m ago

There is no detection method that will prevent AI from accessing systems without also blocking humans. The only thing we can do at this point is throttling.

afro88•2m ago

I'm becoming more and more convinced that a big source of outrage on the internet is caused by people assuming that all other people are a homogenous blob.

It's not that "we" are going from one thing to another. It's that these are two different people, with different ethical boundaries.

bel8•21m ago

Interesting, how is the chromium sandboxed? profile dir cli param? or deeper like chromium engine framework embeded in the application?

dvt•19m ago

Just simply a separate Chromium binary (not your usual browser).

rjtc•13m ago

anyone can spin these kind of side projects and do easy talk, but the moment you actually try to use this on signed-in Linkedin or Amazon its going to fail

the only solution is to drive your regular browser with all your sessions/cookies via an extension

dvt•10m ago

> the only solution is to drive your regular browser with all your sessions/cookies via an extension

This is exactly what I'm doing, but mocking/randomizing all the sessions/cookies/params (like resolution, OS, WebGL , etc.) in a separate Chromium binary. It's popular these days, but imo using your normal browser for agenting stuff is a very bad idea. These models do dumb stuff all the time.

carsoon•6m ago

no it doesn't fail. Agents are very good at comparing traffic characteristics from a real browser and a headless/automation browser and getting it to behave in the same manner. It's a cat and mouse game for sites stopping unauthorized access but right now llm agents are ahead.

For testing proxies should be used to avoid IP ban issues but given enough time modern agents can figure out how to bypass most of the modern scraping/automation prevention mechanisms.

I have built and used a lot of different automations and web scraping implementations for my business and it's never got permanently stuck yet, some take a bit longer, some shorter, but all within a reasonable time with little external help they have succeeded in their tasks.

kurisufag•9m ago

the 4get dev did the same thing a little while ago for his metasearch engine: https://git.lolcat.ca/lolcat/4play

pprotas•6m ago

Camoufox bypasses most blocks with no problems https://camoufox.com/

onion2k•6m ago

(using a custom side-loaded plugin that talks over websockets to a "driver")

Is that necessary? You could start Chromium with an open debug port and use Chrome Devtools Protocol to send commands.

arjunchint•31m ago

their static ip's were initially good and didn't get flagged, but now most sites are recognizing their ip ranges and blocking.

Muse's utility has significantly dropped with the blockages.

To become truly useful again they will need to use residential proxies, but I can't see them use those due to the risks and reputational damage.

sejje•7m ago

they can just use the user ip. i think grok already does this.

Gareth321•3m ago

Most residential proxies are already far more blocked and rate limited than any Meta IP. The internet is becoming a very weird place, where individual and "trusted" personal IPs are becoming a kind of commodity. Some sites are already scoring IPs based on usage activity - like a credit score. It's only a matter of time until this data is collated and commoditised. AI analysis is turning this up to 11.

kevmo314•5m ago

Muse ran into a captcha and asked me if I wanted it to solve it.

So of course I clicked yes and it dutifully convinced the site that it was not a bot.

xnickb•2m ago

For 2(3?) decades we've been training the robots to tell traffic lights from fire hydrants. It's finally paying off.

AI upscaling is coming to PS5

Bryan Cantrill: Keynote (RustConf2026): Rust, in Sickness and in Health [video]

A lack of rockets is creating a crisis for space technology companies

Show HN: Mixdog – open-source coding agent for Windows desktop

Decode what's behind a QR code, useful for retrieving wi-fi passwords

Sukumar Sen and India's First Election

Show HN: Fantasy Walk – A cozy walking RPG for Android

I Don't Remember the Last Time I Checked Mastodon

The hard part of an editable table is knowing what changed

Griffin, the first model to pass the video Turing test

Dotfiles – Mise-En-Place

OpenDots

Stages of grief about the effect of AI on open source

Rust in the kernel? What about Rust without the kernel

Show HN: AviGPT-250M – Edge SLM with NVMe memory bus (488MB, runs offline)

Metadata Reactor – image metadata for 11 platforms

Thoughts on Languages from 2000's

Show HN: Run Google DataProc cluster on your local

The Empire State Building's Mooring Mast

MySQL Workbench 26

Aaron

Infinitrance Live Stream [video]

US tells France and Germany to release diesel stocks or face US export ban

There's a spike in Bitcoin activity around the time foreign aid money goes out

Shimano Bicycle Museum Review

Yes I still use a landline [video]

Ideas on modernizing the open-source desktop

The Comedy of the Commons (1986)

Singapore gov. launches dating app with Gale-Shapley stable matching algorithm

Amazon seeks to offload $8B of Nvidia chips to investors