Ask HN: Is there a service that offers Common Crawl as an API?

3•georgehill•5h ago

I am trying to do some data analysis work. I don't want the full dataset. I want only two things: give me the hostname, and give me all the pages or URLs with their HTML.

Comments

phillipseamore•4h ago

Not that I know of but there are various tools like https://github.com/alwalxed/wayurls

georgehill•2h ago

thank you will check this out

After 16 years, we're renewing the StackOverflow Brand

Everything You Need to Know About Sparkle's Upcoming 24GB Arc Battlemage GPU

End of History – Francis Fukuyama (1992)

5 Steps to N-Body Simulation

Ash (Almquist Shell) Variants

'Just Write Great Content' Is the Worst SEO Advice You'll Hear

So Yeah – I hate useEffect

15 Years of Shader Minification

Apple hits back at US judge's 'extraordinary' contempt order

EM-LLM: Human-Inspired Episodic Memory for Infinite Context LLMs

Model Context Protocol (MCP) Clearly Explained

Show HN: Codigo – The Programming Language Repository

A whippet waypoint / Nofl: A Precise Immix

Mexico sues Google over changing Gulf of Mexico's name for US users

Why should I care? or why punks are correct and old wise philosophers are wrong

Ask HN: How do you hire outside your competencies?

Tesla Worker Knew His Anti-Musk Website Was a Risk. He Did It Anyway

AI in Policymaking, Some Thoughts

Most visitors leave your site without ever talking to you. Fix it with this

Data-hungry browsers: the choice of 90% of users worldwide?

Floating kelp forests have limited protection despite intensifying heat threats

Mill as a Direct Style Build Tool

Frink data file for non-changing units

Airlines Are Collecting Your Data and Selling It to ICE

Warning signs hinted at Spain's unprecedented power outage

NOT a 3 year old chimney sweep (2022)

Apple Turnover

Germania: The Origins and Migrations of the Germanic Peoples

Taking Control of Image Processing

Uncle Bob Martin Morning Bathrobe Rant: Types