Ask HN: Is there a service that offers Common Crawl as an API?

6•georgehill•15h ago

I am trying to do some data analysis work. I don't want the full dataset. I want only two things: give me the hostname, and give me all the pages or URLs with their HTML.

Comments

phillipseamore•14h ago

Not that I know of but there are various tools like https://github.com/alwalxed/wayurls

georgehill•13h ago

thank you will check this out

Ask HN: If 1 person can control 10 AI agents, why would still need that person?

Ask HN: What are good high-information density UIs (screenshots, apps, sites)?

Ask HN: RAG or shared memory for task planning across physical agents?

Ask HN: How much better are AI IDEs vs. copy pasting into chat apps?

Ask HN: Escaping a Low-Paying Nepali IT Job and Ineffective Learning Cycle

Ask HN: What would you do with the #manga chat channel in Libera IRC network?

Ask HN: Is there a service that offers Common Crawl as an API?

Ask HN: Anyone using knowledge graphs for LLM agent memory/context management?

Blazeio.SharpEvent: A Python Async Primitive That Scales to 1M Waiters with O(1)

Ask HN: AI-Filtering Browser Extension?

AI Summarizer: Summarize Web, YouTube and PDFs in Seconds–Free

Ask HN: How to get good at marketing your product and SEO?

Ask HN: Hackathons feel fake now

OSUniverse: Building a Better OSWorld

Ask HN: How could vibe coding show the code at a high level to non-programmers?

Ask HN: Are you using AI coding assistance?

Ask HN: Nvidia GeForce RTX 5060 arrives May 19 at $299 revive PC builds?

Ask HN: How do you obtain software development contracts?

Getting tired of Helm – any better way to handle deployments in Kubernetes?

Ask HN: Did Aliexpress stop shipping to US?

Why do websites prevent pasting via onpaste="return false;"

Ask HN: Help us validate our idea of an administrative app for small businesses

Ask HN: How are you managing LLM inference at the edge?

We built an AI-powered voice tool to boost sales

Ask HN: Why is the sender chat box always on the right?

Is a Smaller Internet Better?

Ask HN: Which Firefox add-ons are you using in 2025?

Ask HN: What's the best framework for building Mac/Windows desktop apps in 2025?

Ask HN: Have you used Claude Code? Is it any good?

Ask HN: Has anyone managed to pass Meta's Access Verification?

Ask HN: Is there a service that offers Common Crawl as an API?

Comments

Ask HN: If 1 person can control 10 AI agents, why would still need that person?

Ask HN: What are good high-information density UIs (screenshots, apps, sites)?

Ask HN: RAG or shared memory for task planning across physical agents?

Ask HN: How much better are AI IDEs vs. copy pasting into chat apps?

Ask HN: Escaping a Low-Paying Nepali IT Job and Ineffective Learning Cycle

Ask HN: What would you do with the #manga chat channel in Libera IRC network?

Ask HN: Is there a service that offers Common Crawl as an API?

Ask HN: Anyone using knowledge graphs for LLM agent memory/context management?

Blazeio.SharpEvent: A Python Async Primitive That Scales to 1M Waiters with O(1)

Ask HN: AI-Filtering Browser Extension?

AI Summarizer: Summarize Web, YouTube and PDFs in Seconds–Free

Ask HN: How to get good at marketing your product and SEO?

Ask HN: Hackathons feel fake now

OSUniverse: Building a Better OSWorld

Ask HN: How could vibe coding show the code at a high level to non-programmers?

Ask HN: Are you using AI coding assistance?

Ask HN: Nvidia GeForce RTX 5060 arrives May 19 at $299 revive PC builds?

Ask HN: How do you obtain software development contracts?

Getting tired of Helm – any better way to handle deployments in Kubernetes?

Ask HN: Did Aliexpress stop shipping to US?

Why do websites prevent pasting via onpaste="return false;"

Ask HN: Help us validate our idea of an administrative app for small businesses

Ask HN: How are you managing LLM inference at the edge?

We built an AI-powered voice tool to boost sales

Ask HN: Why is the sender chat box always on the right?

Is a Smaller Internet Better?

Ask HN: Which Firefox add-ons are you using in 2025?

Ask HN: What's the best framework for building Mac/Windows desktop apps in 2025?

Ask HN: Have you used Claude Code? Is it any good?

Ask HN: Has anyone managed to pass Meta's Access Verification?