frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

Production tests: a guidebook for better systems and more sleep

https://martincapodici.com/2025/05/13/production-tests-a-guidebook-for-better-systems-and-more-sleep/
78•mcapodici•1y ago

Comments

ashishb•1y ago
Here's a general rule that I follow along with this and that is "write tests along the axis of minimum change"[1]. Such tests are more valuable and require less maintenance over time.

1 - https://ashishb.net/programming/bad-and-good-ways-to-write-a...

compumike•1y ago
I'd add that, in terms of tactical implementation, production tests can be implemented at least two different ways:

(1) You set up an outside service to send an HTTP response (or run a headless browser session) every minute, and your endpoint runs some internal assertions that everything looks good, and returns 200 on success.

(2) You set up a scheduled job to run every minute internal to your service. This job does some internal assertions that everything looks good, and sends a heartbeat to an outside service on success.

For #2: most apps of any complexity will already have some system for background and scheduled jobs, so #2 can make a lot of sense. It can also serve as a production assertion that your background job system (Sidekiq, Celery, Resque, crond, systemd, etc) is healthy and running! But it doesn't test the HTTP side of your stack at all.

For #1: it has the advantage that you also get to assert that all the layers between your user and your application are up and running: DNS, load balancers, SSL certificates, etc. But this means that on failure, it may be less immediately clear whether the failure is internal to your application, or somewhere else in the stack.

My personal take has been to lean toward #2 more heavily (lots of individual check jobs that run once per minute inside Sidekiq, and then check-in on success), but with a little bit of #1 sprinkled in as well (some lightweight health-check endpoints, others that do more intense checks on various parts of the system, a few that monitor various redirects like www->root domain or http->https). And for our team we implement both #1 and #2 with Heii On-Call https://heiioncall.com/ : for #2, sending heartbeats from the cron-style check jobs to the "Inbound Liveness" triggers, and for #1, implementing a bunch of "Outbound Probe" HTTP uptime checks with various assertions on the response headers etc.

And this production monitoring is all in addition to a ton of rspec and capybara tests that run in CI before a build gets deployed. In terms of effort or lines of code, it's probably:

    90% rspec and capybara tests that run on CI (not production tests)
    9% various SystemXyzCheckJob tests that run every minute in production and send a heartbeat
    1% various health check endpoints with different assertions that are hit externally in production
And absolutely agree about requiring multiple consecutive failures before an alarm! Whenever I'm woken up by a false positive, my default timeout (i.e. # of consecutive failures required) gets a little bit higher :)
hugs•1y ago
yeah, full end-to-end tests/monitors are like fire alarms: they can often tell you something is wrong, but not exactly what is wrong. but that doesn't mean fire alarms have no value. most common failure mode for teams are having too many or none at all. but having a few in a few key places is the way to go.
mhw•1y ago
The fabulous blazer gem includes a feature for #2: https://github.com/ankane/blazer?tab=readme-ov-file#checks - it’s limited to checks that can be expressed as SQL queries, but that can get you quite a way
aleksiy123•1y ago
At Google we call these probers.

Does anyone know of any tools/saas that do this.

Was thinking it may be a good potential product.

Especially if it was super easy to generate/spin up for side projects.

hugs•1y ago
"testing in production" can be controversial, but this is a well-balanced take on it.

lately i've been working on a decentralized production testing network called 'valet network' [1] (full-disclosure: selenium creator here)

i suspect production tests are the killer app for this kind of network: test any site on a real device from anywhere on idle devices that more closely match real world conditions, but as mentioned in the article, it's not that simple. dev users will still need to be smart about creating test data and filtering out the tests from system logs. i'm still in the "is this something people want?" learning phase, even though this is definitely something i want and wish i had when i was helping to fix healthcare.gov back in 2013/2014.

[1]: https://gist.github.com/hugs/7ba46b32d3a21945e08e78510224610...

vasusen•1y ago
Thank you for the balanced take on an extremely spicy topic.

At WePay (YC S09) we debated this extensively and came up with a similar middle of the way solution. Making sure that a credit card can get tokenized is the critical flow and should run every minute. We ended up with about 4-5 very quick production tests. They helped with debugging as well as alerting.

I am now building a full, automated testing solution at Donobu (https://www.donobu.com), and production tests definitely come up as their own subcategory of e2e tests. I am going to use your guidelines to refine our prompt and bound our production test generator.

testthetest•1y ago
> Running a test every minute, or 1440 times a day, will show up quite a lot in logs, metrics, and traces.

...not to mention that automated tests are by definition bot traffic, and websites do/should have protections against spam. Cloudflare or AWS WAF tends to filter out some of our AWS DeviceFarm tests, and running automated tests directly from EC2 instances is pretty much guaranteed to be caught by Captcha. Which is not a complaint: this is literally what they were designed to do.

A way to mitigate this issue is to implement "test-only" user agents or tokens to make sure that synthetic requests are distinguishable from real ones, but that means that our code does something in testing that it doesn't do in "real life". (The full Volkswagen effect.)

burnt-resistor•1y ago
Also known as deep monitoring: checking that functionality is available and working correctly.

The Secret ThinkPad Powerbutton Code to Bring Dead Laptops Back to Life (2007)

https://masnick.com/posts/the-secret-thinkpad-powerbutton-code-to-bring-dead-laptops-back-to-life/
193•sammatilda•5h ago•79 comments

Many of Amazon’s Black Friday and Cyber Monday “Deals” Offered No Savings

http://groundworkcollaborative.org/news/amazons-decoy-deals-groundwork-investigation-reveals-near...
367•DeepLogin•3h ago•201 comments

How to make almost anything

https://pub.cba.mit.edu/863.26/site/
652•o4c•11h ago•106 comments

Nvidia Was in Trouble Before the Jensen Huang Mess

https://www.wired.com/story/nvidia-was-already-in-trouble-before-the-jensen-huang-mess/
61•Brajeshwar•1h ago•35 comments

Universal XSS in uBlock Origin from website-driven element picker

https://iter.ca/post/ubo-uxss/
537•zaik•14h ago•154 comments

Interpreting claims that a theorem is correct because it was checked in Lean

https://xenaproject.wordpress.com/2026/11/12/and-the-proof-is-checked-in-lean/
200•matt_d•9h ago•74 comments

What is w2rt6byl5nf1opmc-a.com?

https://jack.cab/blog/siterate
256•dverlaeckt80•3h ago•109 comments

OOS – an open operating system for the Rigol Scopes

https://www.eevblog.com/forum/testgear/oos-an-open-operating-system-for-the-rigol-scopes/
77•qwezxcrty•2d ago•26 comments

The Active Directory Era Is Finally Over

https://infisical.com/blog/active-directory-machine-identity
15•FinnLobsien•2h ago•0 comments

Researchers track whole body movement with just a smartwatch

https://news.cornell.edu/stories/2026/11/researchers-track-whole-body-movement-just-smartwatch
149•PaulHoule•2d ago•46 comments

Qwen4-Max

https://qwen.ai/blog?id=qwen4-max
770•ilreb•14h ago•390 comments

What's Next for Nvidia?

https://steveblank.com/2026/11/15/whats-next-for-nvidia/
401•zdw•17h ago•506 comments

Old Bitly links are being hijacked

https://ransomnews.com/burger-king-tweet-bitly-link-hijack-2026/
351•hspeiser•18h ago•181 comments

Coffeehouses and the Rise of Science [pdf]

https://www.monirbounadi.com/Bounadi_JMP.pdf
142•Michelangelo11•5h ago•82 comments

Winudf – DVD / BluRay driver for DOS, Windows 1.x – 3x / 95 / 98 / ME

https://github.com/ages2001/WINUDF
63•Oxodao•8h ago•7 comments

Nature-Based Sanitation. Cihuatlán: A Large-Scale Artificial Wetland

https://revista.unaminternacional.unam.mx/en/nota/12/saneamiento-basado-en-la-naturaleza-cihuatla...
6•bryanrasmussen•2d ago•1 comments

Show HN: ThingStudio, Node-RED style programming for microcontrollers

https://thingstudio.net/
27•mkarliner•5d ago•3 comments

Protomaps: Self-hosting map tiles has gotten cheap and easy

https://bruceroettgers.eu/blog/protomaps/
433•bcye•21h ago•77 comments

Tor Project Issues Rapid-Response DoS-Defense Fix Amid Sustained DDoS Attack

https://forum.torproject.org/t/security-release-0-4-9-17/22454
219•aillia•16h ago•56 comments

Handwriting, as a Font

https://ductus.sorenblank.com
408•coinfused•21h ago•70 comments

Preserving 30 years of Salon journalism

https://blog.archive.org/2026/11/12/preserving-30-years-of-salon-journalism/
194•CharlesW•17h ago•55 comments

The Lasting Influence of John Berger

https://www.newyorker.com/magazine/2026/11/16/john-berger-a-life-tom-overton-book-review
53•prismatic•4d ago•4 comments

A short explainer on how El Niño works [video]

https://www.youtube.com/watch?v=7_JF3etiAv8
285•nharada•22h ago•169 comments

Crows make decisions faster than we can

https://www.nytimes.com/2026/11/15/science/birdbrain-crows-birds-research-decision.html
286•gumby•23h ago•193 comments

Do we still have to denormalize? Have your schema and search it too

https://www.paradedb.com/blog/joins
109•philippemnoel•3d ago•35 comments

Exact Gradients Through a BRep Kernel

https://muni.works/research/vcad
38•ecto•2d ago•12 comments

The Opportunivore's Dilemma

https://www.vqronline.org/essays/opportunivores-dilemma
9•NaOH•2d ago•10 comments

WebGL Flight Simulator with seed generated terrain

https://baron.by.dai.studio/
217•ranger_danger•20h ago•55 comments

Whistleblower accuses Pentagon over Iran strike on US base that killed 6

https://www.cnn.com/2026/11/15/politics/army-soldier-whistleblower-complaint
176•firefax•2h ago•49 comments

A quest to create the perfect whiskey glass

https://www.nytimes.com/2026/11/13/dining/drinks/kurt-josef-zalto-whiskey-glass.html
86•Kaibeezy•2d ago•93 comments