Does this skirt the robots.txt by chance? Not being to fetch any web page is really bugging me and I'm hoping to use a better web_fetch that isn't censored. I'm just going to copy/paste the content anyway.

I know this is pointing to the GH repo, but I’d love to know more about why the author chose to build it this way. I suspect it keeps costs low/free. But why CF workers? How much processing can you get done for free here?

I’m not sure how you could do much more in a CF worker, but this might be too simple to be useful on many sites.

Example: I had to pull in a docs site that was built for a project I’m working on. We wanted an LLM to be able to use the docs in their responses. However, the site was based on VitePress. I didn’t have access to the source markdown files, so I wrote an MCP fetcher that uses a dockerized headless chrome instance to load the page. I then pull the innerHTML directly from the processed DOM. It’s probably overkill, but an example of when this tool might not work.

But — if you have a static site, this tool could be a very simple way to configure MCP access. It’s a nice idea!

From what I can see, if the content I want to enrich is static, the web fetch tool seems sufficient. Is this tool capable of extracting information from dynamic websites or sites behind login walls, or is it essentially the same as a web fetch tool that only works with static pages?