Discussion (72 points, 5 hours ago, 51 comments) https://news.ycombinator.com/item?id=45814461

In my opinion the whole agents.txt thing is bullshit and/or politeness, and should have zero legal significance.

If something is openly available on the internet you should be able to crawl it, and it is the server's responsibility to identify, authenticate and/or ban clients that do not adhere to it's requirements. If Amazon states "You must not use agents" in it's website terms of service then it would be the individual perplexity user not perplexity itself that is breaking it's terms of service, since they are the one operating the agent.

All in all lame move by Amazon.