There (1) is more context on this topic, where author of LLMs.txt idea said 'llms.txt files have nothing to do with crawlers or big LLM companies. They are for individual client agents to use.'

1. https://news.ycombinator.com/item?id=47058870