1. Test pass rate is 99.98% is not good - the only acceptable rate is 100%.
2. Tests should not be quarantined or disabled. Every flaky test deserves attention.
rkomorn•1h ago
On top of 2., new tests should be stress-tested to make sure they aren't flaky so that the odds of merging them go down.
lionkor•38m ago
I can run flaky tests on my machine a thousand times without failure, whereas they fail in CI sometimes.
lab14•57m ago
a test pass rate of 100% is a fairy tale. maybe achievable on toy or dormant projects, but real world applications that make money are a bit more messy than that.
YetAnotherNick•5m ago
Even something as simple as docker pull fails for 0.02% of the time.
IshKebab•1h ago
> Every commit to main triggers an average of 221 parallel jobs
Jesus, this is why Bazel was invented.
simianwords•1h ago
interesting that they have an agent that is triggered on flaky CI failures. but it seems far too specific -- you can have pull request on many other triggers.
there doesn't seem to be any upside on having it only for flaky tests because the workflow is really agnostic to the context.
elteto•27m ago
I think this is the first article that truly gave me “slop nausea”. So many “It’s not X. It’s Y.” Do people not realize how awful this reads? It’s not a novel either, just a few thousand words, just fucking write it and edit it yourself.
zeristor•19m ago
I'm guessing they have a workflow for blog posts, with 100k workflows I was wondering something seems a bit weird.
zX41ZdbW•1h ago
1. Test pass rate is 99.98% is not good - the only acceptable rate is 100%.
2. Tests should not be quarantined or disabled. Every flaky test deserves attention.
rkomorn•1h ago
lionkor•38m ago
lab14•57m ago
YetAnotherNick•5m ago