One of the surprising things about converting HTML to Markdown: You often don't need to.
- grab .md files from repos
- append `/index.md` or `.md` to URL
- use 'Accept: text/markdown' header
- GitHub docs article API -> `/api/article?pathname=`
- ... and lots of other shortcuts
You can try out:
- the playground https://curl.md/playground
- do `curl curl.md/example.com` in your terminal
- or open curl.md/example.com in your browser
GitHub if anyone wants to take a look https://github.com/wevm/curl.md
bhagyeshsp•1h ago
If I understand correctly, my agent just needs to run `curl curl.md/targetsite.com` and it will receive the webpage in an optimized Markdown file format.
Is there any specific depth until which curl.md will crawl?
meagher•53m ago
Yes, your agent can use it with `curl` or via the CLI, SDK, or agent plugins (currently Claude, OpenCode, Pi, Amp are supported). More info https://curl.md/docs/install
> Is there any specific depth until which curl.md will crawl?
No crawling yet. Focused on single page for now.