I've been scraping 241 UK council planning portals – 2.6M decisions so far

UK planning data is technically public. In practice it's locked behind 400+ different council portals, some still running bespoke ASP.NET that looks like it dates from 2004, some behind AWS WAF, all with subtly different schemas. I've spent four months scraping them. I'm now at 241 councils and 2.6 million decisions across England, Scotland and Wales.

The scraping problem

Most UK councils run one of a handful of portal systems, Idox being the most common. In theory this makes things easy. In practice every council has configured theirs differently, some block non-browser requests via TLS fingerprinting, some have rate limits that will get you banned inside 10 minutes, and a handful are running the aforementioned bespoke ASP.NET.

I ended up writing several scrapers: a standard requests-based one, a Playwright-based one for councils that block anything that doesn't look like a real browser, and a curl_cffi one for TLS fingerprinting. Some councils I still can't get. Liverpool's portal sits behind AWS WAF with a JavaScript challenge. I have a working Playwright-based scraper that solves the challenge once and reuses cookies, but the WAF rate-limits the IP after about 10 requests and then blocks me for a day. So I have 60k Liverpool decisions from an old scrape and no easy way to add more.

What I found

The approval rate stuff is what most people come for. Nationally it's around 88%, but it varies wildly by ward within a council, not just between councils.

The more interesting finding came from the time-to-decision data. Across 119 English and Welsh councils, 36.5% of home extension applications missed the statutory 8-week target in 2025, up from 27.9% in 2019. Guildford is the worst at scale: 66% of decisions over target, averaging 13.3 weeks.

What it is now

A postcode checker (free) and paid PDF reports (£19/£79). Zero paying customers so far, which is fine. I've been heads down on data quality and coverage.

Site is planninglens.co.uk if you want to poke around. AMA on the scraping side – that's where the interesting problems are.

Show HN: I built a way to see if your SDK is AI-friendly

Building a Threadiverse Community Platform

Australia threatens tech companies with 2.25% tax if they don't pay publishers

How Do Perpetual Futures Differ from Spot Trading in Crypto?

Meta prepares to undo acquisition of Singapore-based Manus after China ban

Freelancer for hire – full stack, ML, DevOps

Talos OS images are now bit-by-bit reproducible

I Use AI in 2026

Come From

Steal Claude Code Architecture

How to build advanced features for AI chatbots on SSE

Show HN: VibeBrowser – Give your AI agent your real logged-in browser via MCP

Show HN: Financial Database API for Vibe Coders

Hotta GameDriverX64.sys shipping in Neverness to Everness preload

Anthropic Claude Code HERMES.md billing flaw

Scraping 241 UK council planning portals – 2.6M decisions so far

Show HN: BeVisible.app - Blog that runs itself

Xiaomi MiMo Orbit: 100T Token Grant for Builders

SwiftBash: Pure-Swift, sandboxed bash interpreter

Text Is the New Binary

Bugs in the original 1977 Cave Adventure Fortran source

A case report of someone who self-managed Fatal Familial Insomnia

Asimov v1: Open-Source Humanoid Robot

I built a coach for people who are tired of being yelled at by Stockfish

Set a Meeting Budget

Ask HN: When might we not have to do laundry or fold clothes or cook

Google signs classified AI deal with Pentagon

The 278k language running 20% of the Internet

Unitree G1 humanoid robot roller skating [video]

Humanoid robots to become baggage handlers in Japan airport experiment

CrowdStrike Linux Agent Easy way to make it better

Comments