Ask HN: Web scraping in production?

3•arkmm•3h ago

Are any of you maintaining any web scrapers in production?

I've done some for side projects, automated testing, and personal scripts (checking personal bank balances, getting a Global Entry interview slot, etc.), but it always feels very brittle.

Curious what applications people have in industry and what sorts of techniques people use for reliability.

Comments

sargstuff•3h ago

excel web scraping[0] (vs. using python[1] and/or odbc/delimited files)

A few 2025 use cases [2],[3]:

   Use publically available database information (construction, taxes, sales, traffic report, proposed building/zone changes, etc) to find out what's going on withing an area aka. zip code, housing area, 'vacation spot', etc

----

   creative take on topic:

      modern looming / static 'threaded' approach : https://news.ycombinator.com/item?id=43977384

      Structurally reprogrammable magnetic maetamaterials hold promise for biomedicine, soft robotics. ("web" support formed via scraping material in relevant patterns) : https://techxplore.com/news/2025-05-reprogrammable-magnetic-metamaterials-biomedicine-soft.html

      3d printed smart-fabrics : https://techxplore.com/news/2025-05-d-smart-fabrics-flexibility-ability.html

----

[0] : excel scraping : https://www.youtube.com/watch?app=desktop&v=6coVzIt93vk

[1] : python scraping : https://www.youtube.com/watch?v=Oo8-nEuDBkk

[2] : https://dataforest.ai/blog/top-web-scraping-use-cases

[3] : https://www.parsehub.com/blog/web-scraping-examples/

arkmm•3h ago

Neat - didn't realize there were affordances for scraping in Excel (but in hindsight I shouldn't be surprised).

I didn't follow the connection between modern looming and scraping though?

sargstuff•3h ago

hint: silk spider webs & fabric threads.

Guess 3d printing should have been clarified as linear, fused deposition. Melted plastic line gets scraped along plate/material.

The 3d printed web reference, in this instance, being the in-fill pattern. [0]

Robotic metal pinching / incremental sheet forming might be bit more clearer example. [1]

-----

[0] : in-fill pattern : https://jlc3dp.com/blog/choosing-the-right-infill-structure-...

[1] : https://www.youtube.com/watch?v=Jc16Ob-yoDs

9d•3h ago

Scraping is inherently brittle, but it can be very useful for short-term scraping in very specific circumstances. I haven't had any in maybe 10 years.

sargstuff•3h ago

IMHO, "untyped" format/delimited file yes. Directly placing/'compiling' in appropriate topological construct/environment works wonders. aka environment of database, spreadsheet, "reports" with information beyond raw data, etc

Major world system dust particle kalpas

Map of Palaeohispanic Coins and Inscriptions

Cognitronics and the Longest Running Voice in Telephony [video]

IterGen: Iterative Semantic-Aware Structured LLM Generation with Backtracking

ROCODE: Integrating Backtracking Mechanism and Program Analysis in LLMs for Code

SRLCG: Self-Rectified Large-Scale Code Generation, CoT, Dynamic Backtracking

AI #115: The Evil Applications Division

"How did porcelain go from a rare luxury to a commodity?"

Silicon Valley Braces for Chaos

The Myth of the Poverty Trap

Less meat is nearly always better than sustainable meat

Ibcs-us – Linux User space emulation of SCO, Solaris and others

What I Worked on (2021)

Intel's new CEO 'isn't thinking about changes'

Ask HN: Take over FB group when Admin dies?

Good luck to everyone applying for YC summer 2925 batch

How to See MCP in Action?

Why do NYC drivers waste two hundred million hours a year circling the block?

Deploying Software: A Technology Explainer and a Look Toward the Future

Your fingers wrinkle the same way every time you're in the water too long

NeXTSTEP: The Visionary OS, Steve Jobs's Apple Exodus, and the GNUStep Legacy

Nutpie: High-Performance Bayesian Inference

Embedding files in Go using the "embed" package

Quay.io Push Unavailable

Disputing My Block at Justapedia

Women in the Age of Polar Exploration

Ask HN: Should bots actively be banned on HN

JavaScript's New Superpower: Explicit Resource Management

There is nothing revolutionary in women abandoning women's rights

Deep Laziness

Major world system dust particle kalpas

Map of Palaeohispanic Coins and Inscriptions

Cognitronics and the Longest Running Voice in Telephony [video]

IterGen: Iterative Semantic-Aware Structured LLM Generation with Backtracking

ROCODE: Integrating Backtracking Mechanism and Program Analysis in LLMs for Code

SRLCG: Self-Rectified Large-Scale Code Generation, CoT, Dynamic Backtracking

AI #115: The Evil Applications Division

"How did porcelain go from a rare luxury to a commodity?"

Silicon Valley Braces for Chaos

The Myth of the Poverty Trap

Less meat is nearly always better than sustainable meat

Ibcs-us – Linux User space emulation of SCO, Solaris and others

What I Worked on (2021)

Intel's new CEO 'isn't thinking about changes'

Ask HN: Take over FB group when Admin dies?

Good luck to everyone applying for YC summer 2925 batch

How to See MCP in Action?

Why do NYC drivers waste two hundred million hours a year circling the block?

Deploying Software: A Technology Explainer and a Look Toward the Future

Your fingers wrinkle the same way every time you're in the water too long

NeXTSTEP: The Visionary OS, Steve Jobs's Apple Exodus, and the GNUStep Legacy

Nutpie: High-Performance Bayesian Inference

Embedding files in Go using the "embed" package

Quay.io Push Unavailable

Disputing My Block at Justapedia

Women in the Age of Polar Exploration

Ask HN: Should bots actively be banned on HN

JavaScript's New Superpower: Explicit Resource Management

There is nothing revolutionary in women abandoning women's rights

Deep Laziness

Ask HN: Web scraping in production?

Comments