frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

Websites are closing their doors to AI agents

https://www.theverge.com/ai-artificial-intelligence/1013195/ai-agents-blocked-web
45•evolve2k•1h ago

Comments

spwa4•1h ago
The whole web is based on selling people's attention, providing free services as "payment" to people. The whole value proposition of AI is to give the services without giving people's attention?
cyanydeez•57m ago
Not "give" take.

Look at spotify and their royalties. Theres already such little value in human based internet that agents will race to the bottom.

The only internet content will be worse than adware. Itll just be agent poison meant to take your money.

Any other viewpoint seems to ignore the already shattered shared internet.

mvkel•54m ago
Most content has been behind a paywall / login wall for years
bdcravens•49m ago
Solvable problems (just login or pay). This is more than just blocking anonymous scrapers. Sites are blocking access to agents even when they have a proper "ticket" for access.
CamperBob2•36m ago
Sites are blocking access to agents even when they have a proper "ticket" for access.

(Shrug) Well, they must not want to sell anything to me.

This is not a battle that site owners want to fight. They may think it is, but they're horrifically wrong about that.

CuriouslyC•16m ago
It's insane to me that people don't see that agents are going to be how you interact with your computer going forward, almost exclusively. The people who make things hard for agents are going to get eviscerated by the people who don't just make it easy for agents to do things, but market primarily to agents rather than humans.
canucktrash669•54m ago
Entire networks. GPT dot became useless as it got systematically blocked by Cloudflare, Google cloud, vercel, etc. At the same time OpenAI blocked solutions like tailscale and others effectively killing the utility of browser control by dot.
canucktrash669•53m ago
It’s still okay at email.
variety8675•53m ago
Companies don't want it to be easy to cross shop their competition, the whole premise of Amazon is that it's so easy and simple you don't even check the competition. AI agents are leveling the playing field and helping customers find the best deals, so of course they're being banned
canucktrash669•51m ago
Agents are likely the slow death of Amazon. Muse could decide to sell anything and group buy their way into the everything store.
criddell•38m ago
Current AI agents overindex on price. The price has to be reasonable but I often rank fast shipping and easy returns higher than price.
xnx•21m ago
True, and agents can also present a comparison table of options including: total price, shipping time, return convenience, etc.
evmar•12m ago
Recently bought something online from Home Depot and it arrived damaged. When I asked them for help they said I had to drive it back to the store myself.

If I have to drive to the store to get it, I wouldn’t have shopped with them! It feels like every time I don’t order with Amazon the stores remind me why I ought to.

alansaber•37m ago
versooth•51m ago
This is opportunity.
BorisMelnik•48m ago
my prediction: ai agents will have their day, but not on this version of the web. websites will have a separate xml/json/yaml version that agents can peruse without all the ads/js/etc
morkalork•34m ago
Without the ads? The pessimist in me thinks with all the ads, prompt injections, and long-form content marketing included. Already you can see Google's AI results rehashing marketing slop in the answers about how amazing one product or another is.

Wasn't there some restaurant review site that gets accused of doing pay-to-play hiding of negative reviews and promoting positive ones that calls up businesses and basically extorts them into their services? They have all the incentive in the world now to manipulate the results shown to AI agents. You won't even see the manipulation directly yourself when there's an AI sitting between you and the source acting like an obfuscation layer.

tracerbulletx•18m ago
Why would that happen if the website owners are explicitly trying to prevent access? This isn't a technical issue, the agents are perfectly capable of working with the existing websites regardless of ads and javascript. The semantic web and machine readable representations was a 90s thing, agents don't need that really.
zzzeek•11m ago
the js/yaml version would require a license key that connects to a business arrangement between the website and the owner of the crawler. the plain HTML version would adversarially block them (or even identify them, where they could be sued...once we have legal precedents for this stuff).
whimsicalism
xnx•44m ago
Without a convenience moat, commerce will get even more adversarial: membership pricing, surveillance pricing, loyalty perks, varied products to avoid price comparison (like mattresses and Costco).

Fortunately, if AI is really superhuman in capabilities it may still win the battle for the customer and keep prices competitive: aggressive price comparison, alternatives discovery, negotiation, group buying, etc.

haunter•36m ago
Like Reddit is shutting down the RSS feeds :(

I mean there are multiple reasons not just the AI agents but that's also a part of it

Jcampuzano2•32m ago
This is expected. A lot of these companies that are closing their doors make a large portion of their revenue based off of ads and upsells which agents can easily be instructed to ignore, and don't interact with like a human.

Another more blunt way to say it is that people can instruct their agents to look out for their best interests (and while there are cracks they continue to get better at this), and companies hate that.

People will work around it similar to how we've worked around with adblockers for years, and it will just become a game of attrition as these companies work to preserve their moats until they find a way to preserve this revenue stream off of people's agents.

dawnerd•24m ago
It’s also a “you can’t scrape us but we can scrape you” mentality these companies are getting into.

It really sucks for small businesses. Their websites get absolutely hammered and they can’t afford or don’t know how to stop it. Cloudflare ends up winning as they’re the easiest solution.

m-p-3•20m ago
We host a website in Canada (physical server hosting VMs in a Canadian DC) and with the ongoing trade-war we do whatever we can to avoid anything US-based tech such as Cloudflare. So far we use Anubis to protect our services from scraping and it's been a decent experience.
zzzeek•12m ago
it looks like this is solving itself in the usual and productive way:

> Instead of trying to fight Amazon, the company has partnered with retailers like Walmart, Gap, Sephora, Best Buy, and others, allowing Muse to shop on their sites. At the same time, Shopify has also opened its network of storefronts to AI agents.

they're making deals with the sites so that the agents can work, the sites get compensated. I hope for this "AI can do what it wants if it pays" model to expand further. Cloudflare told us months ago we'd have an option to get paid by AI crawlers. I signed up to join but of course as a nobody I never got a reply.

BatchJob•11m ago
This is just like all tech industry disruption.

First they forced their malware bot farms on us, now they want to rent them to us so they can turn the entire web into the dark web.

And eventually rug pull off the major retail sites and just redirect traffic to their own darker sites to scrape commissions, sell direct and keep the change.

I'm sure the teams thinking this up buy their lunches from a ghost kitchen run by illegals that's never had a health inspection,

located in the back of a laundromat with a phony business license and website that only accepts crypto.

It's really hard to build a walled garden just for human users, and not for agents. Have to pick a side.
dasil003•12m ago
Your first sentence makes sense, but not following your second. Who needs to pick and what are the sides they are choosing between?
•
15m ago
hm. for me, AI agents effectively already have their day unless there is a retrenchment. i essentially do not interact with the web (outside of a few social media sites) anymore
wpm•12m ago
My agents can just take that feed and render it into a custom front end just for me too. Why would I browse the adversarial, bloated, ad infested version?
likium•9m ago
Isn't that just API? We had them. Reddit/Zillow/Wells Fargo/etc restricted theirs due to business reasons, not technical ones.

Building a Compositor for a Userbase of One

https://onewireout.com/writeups/resonance/
1•airhangerf15•2m ago•0 comments

How to Set Up Multiview with RTSP Cameras Using Red5 Pro?

https://www.red5.net/blog/revolutionizing-live-video-streaming-multiview-with-rtsp-cameras-using-...
1•mondainx•2m ago•0 comments

Safety and Liveliness in Software Engineering

https://en.wikipedia.org/wiki/Safety_and_liveness_properties
1•_josh_meyer_•2m ago•0 comments

AI-Assisted Product Discovery: When Code Is Cheap, Judgment Is the Job

https://age-of-product.com/ai-assisted-product-discovery/
1•swolpers•6m ago•0 comments

Python's binary wheels and supply chain attacks

https://rushter.com/blog/python-binary-wheels/
1•birdculture•8m ago•0 comments

Microsoft drops AI datacentre project in Essex over power supply concerns

https://www.theguardian.com/uk-news/2026/oct/18/microsoft-drops-huge-ai-datacentre-project-in-ess...
3•wondersky•13m ago•0 comments

Benchmarking DuckDB v2.0-dev on Nvidia Vera

https://duckdb.org/2026/10/16/nvidia-vera-benchmark
1•tosh•15m ago•0 comments

Ask HN: System One models for autonomous agents

1•manlymuppet•20m ago•0 comments

Campus deplatformings hit record high

https://expression.fire.org/p/campus-deplatformings-hit-record
3•iamnothere•20m ago•0 comments

Simultaneous high-fidelity single-qubit gates in a spin qubit array

https://www.nature.com/articles/s41567-026-03466-2
1•bookofjoe•20m ago•0 comments

Probabilistic Programming

https://adriansampson.net/doc/ppl.html
1•rramadass•31m ago•1 comments

Casio GG-B100X-1A

https://www.casio.com/uk/watches/gshock/product.GG-B100X-1A/
1•asjkff•34m ago•1 comments

Israeli settlers attack European diplomats, beat French mayor

https://www.channelnewsasia.com/world/israeli-settlers-attack-european-diplomats-beat-french-mayo...
6•Markoff•35m ago•0 comments

macOS 15.8.2 Breaks OpenGL

https://developer.apple.com/forums/verify-human/verify.html?next=/forums/thread/850550
20•Audiophilip•35m ago•7 comments

Show HN: AI Zombie Trend–Generating multi-shot narrative videos from two photos

https://aizombietrend.online/
1•dawn3727•36m ago•0 comments

Show HN: Secret Guard – a local LLM gateway; keys never leave your machine

https://github.com/luochen1990/secret-guard
1•luochen1990•37m ago•0 comments

Snow Leopard Was Peak Computer

https://www.alara.blog/snow-leopard-was-peak-computer/
3•asaddhamani•39m ago•0 comments

Galileo: History and Myth

https://historyforatheists.com/2026/10/galileo-history-and-myth/
1•throw0101a•39m ago•0 comments

Built a task tracker for teams that hate project management

https://turtletask.app/
1•james_mcavoy•39m ago•0 comments

Show HN: Gecit, share a Windows PC's keyboard and mouse with a Mac over QUIC

https://kasparovabi.github.io/gecit/
2•kasparovabi•40m ago•0 comments

Show HN: Wisp – a voice AI on your shoulder, with spatial audio in the browser

https://wispplayer.com/?tab=ai
1•Serg75•43m ago•0 comments

Study shows cannabis is not associated with better sleep quality

https://news.northwestern.edu/stories/2026/10/think-cannabis-helps-you-sleep-better-think-again
4•gmays•44m ago•3 comments

Trust in Depth: Giving agents more autonomy without losing control

https://aws.amazon.com/blogs/machine-learning/trust-in-depth-giving-agents-more-autonomy-without-...
1•burcsahinoglu•45m ago•0 comments

Show HN: BricksChat – Generate Lego builds from images, including Technic

https://brickschat.com/
2•mireklzicar•48m ago•0 comments

Omarchy Gets the Web Browser Launching Faster by Avoiding XDG Overhead

https://www.phoronix.com/news/Omarchy-Faster-Web-Browser
1•mikece•48m ago•0 comments

Negotiation tips for people who hate negotiating

https://coached.com/essays/negotiation-tips-for-people-who-hate-negotiating/
1•rohanm93•48m ago•0 comments

Why we decided to open source Relate

https://viablesystems.ai/writing/why-we-decided-to-open-source-relate
2•dennisy•49m ago•1 comments

Aligning a Personal Site with Indieweb and Web Standards

https://htsh.pl/blog/web-standards-audit/
1•jruohonen•50m ago•0 comments

GhostOS – Lightweight, Offline Desktop AI Assistant (macOS/Win/Linux)

https://github.com/GhostOS-ai/GhostOS
1•GhostOS•53m ago•0 comments

Gnome OS Preview

https://blogs.gnome.org/adrianvovk/2026/10/16/gnome-os-preview/
1•leephillips•54m ago•0 comments