Ask HN: Is there a service that offers Common Crawl as an API?

7•georgehill•9mo ago

I am trying to do some data analysis work. I don't want the full dataset. I want only two things: give me the hostname, and give me all the pages or URLs with their HTML.

Comments

phillipseamore•9mo ago

Not that I know of but there are various tools like https://github.com/alwalxed/wayurls

georgehill•9mo ago

thank you will check this out

pluto_modadic•8mo ago

there's index.commoncrawl.org where you can ask for a domain with wildcards.

Ask HN: Anyone Using a Mac Studio for Local AI/LLM?

Ask HN: Non-profit, volunteers run org needs CRM. Is Odoo Community a good sol.?

Ask HN: Non AI-obsessed tech forums

Ask HN: Ideas for small ways to make the world a better place

Ask HN: 10 months since the Llama-4 release: what happened to Meta AI?

AI Regex Scientist: A self-improving regex solver

Ask HN: Who wants to be hired? (February 2026)

Ask HN: Who is hiring? (February 2026)

Ask HN: Any International Job Boards for International Workers?

Tell HN: Another round of Zendesk email spam

Ask HN: Why LLM providers sell access instead of consulting services?

Ask HN: Is Connecting via SSH Risky?

Ask HN: What is the most complicated Algorithm you came up with yourself?

Ask HN: Has your whole engineering team gone big into AI coding? How's it going?

Ask HN: How does ChatGPT decide which websites to recommend?

Ask HN: Is it just me or are most businesses insane?

Ask HN: Mem0 stores memories, but doesn't learn user patterns

Ask HN: Anyone Seeing YT ads related to chats on ChatGPT?

Ask HN: Is there anyone here who still uses slide rules?

Ask HN: Does global decoupling from the USA signal comeback of the desktop app?

Kernighan on Programming

We built a serverless GPU inference platform with predictable latency

Ask HN: How Did You Validate?

Ask HN: Does a good "read it later" app exist?

Ask HN: Have you been fired because of AI?

Ask HN: Cheap laptop for Linux without GUI (for writing)

Ask HN: Anyone have a "sovereign" solution for phone calls?

Test management tools for automation heavy teams

Ask HN: OpenClaw users, what is your token spend?

Ask HN: Has anybody moved their local community off of Facebook groups?