I understand wanting to get money, but honestly, there is no way I would give money to this website in it's current state, you are giving me far too little info before asking me to hand over a credit card.

Then, if someone gives you £19, a crazy amount of money honestly, the last page of the report is an advert to give them 4 times more!

It would be good to add appeal data in (also a public gateway) to show which councils are just being unreasonable.

I personally think the planning regulations in this country are the cause of many ills, including the housing shortage. It just costs so much to get through planning these days, it is often just not worth it. Data like this could help us get that changed.

> You may not use automated tools to scrape, copy, or bulk-download data from our service.

Pot kettle, huh.

For the more challenging scrapes, would highly recommend using the Chrome Devtools MCP to be able to attach the network requests, being made by the browser to the site, as context for your agent/LLM chat - this approach really helped me to write a solid API-based scraper (also using curl_cffi) and bypassed the old tedious playwright-based approach I used to rely on.

Careful not to expose the councils too publicly before they shut you off

There's a Royal Institute of Town Planners, they probably have a magazine you could advertise in (but equally that might get you blocked, idk).

RICS people could probably use the data too? I guess it's useful house-buyer info; houses in the vicinity had successful loft conversions, say.

On the data side - it's something of a moat for you now, but I could see you being successful with FOI requests. An MP might be interested in championing open data access.

I appreciate that won't necessarily capture live / recent data. But it might be quicker than waiting for rate-limits to reset.

> UK planning data is technically public.

it's public, but still copyrighted by those who submitted it

the councils also have database rights over their database, unless you've obtained explicit permission from them directly

https://en.wikipedia.org/wiki/Database_right#United_Kingdom

> I ended up writing several scrapers: a standard requests-based one, a Playwright-based one for councils that block anything that doesn't look like a real browser, and a curl_cffi one for TLS fingerprinting.

so they're explicitly trying to stop you doing this, and ... you're openly admitting to bypassing their technical measures to try and stop you?

have you heard of the Computer Misuse Act?

I doubt the 240 councils are going to be happy once they find out you've done this, especially if you're selling it on for profit

See also the open addresses project by Data Adaptive [1] which is using Freedom of Information requests to publish public council tax address data. The problem they have run into there is that their address datasets are derived from proprietary Ordnance Survey data.

It looks like data.gov.uk is in the process of standardising the planning application process, and publishing them under OGL [2].

[1]: https://www.owenboswarva.com/blog/post-addr44.htm [2]: https://www.planning.data.gov.uk/dataset/planning-applicatio...

I did a search for my postcode and got given results for a different area and council miles away

1. Brilliant! Governments (and corps) treat public data like it’s theirs not ours. Information yearns to be free.

2. Having said that, you are likely violating T&Cs by scraping at all.

3. It is a lot easier to defend your position if you are making it free and public yourself.

4. But paying for food is nice

5. I suggest the business model here is providing architects and lawyers with strong evidence of prior planning decisions nationally

Most people applying for (difficult) planning have experience locally. But the planning system is a mess because it is not coherent nationally or regionally. The win here is not providing a copy of your data (that has legal issues) but providing pointers to decisions that support the case of the person paying you.

So I want to turn an old pub into tasteful housing and a cafe for the local village. The local planning team don’t like it, I could spend money bribing them and the councillors (see how much I understand British democracy) or I could get from you the fifteen pub to housing conversion decisions from around the country and use that to help my bribed councillors defend their u-turn

Everyone wins :-)

If you really want the data, just FOI it for goodness' sake.

I get the distinct impression that many of these outfits aren't really advocating for impoved transparency but are simply trying to exploit and monetise illicitly obtained government data to make a quick buck.