Ask HN: Are your code benchmarks statistically valid?
1•DenisM•9mo ago
It appears that some benchmarking tools provide average or percentile values but no information about statistical validity of the result. What did you observe or do in your career?
Comments
Magma7404•9mo ago
I haven't got a clue since I have never used any benchmark tool in my whole life. Could you tell us in which professional context you would use that kind of tool?
DenisM•9mo ago
Run a benchmark to compare performance of my product before and after making a change in code.
gus_massa•9mo ago
I run it 5 times and drop the two extremes. If the middle tree don't overlap, I'm probably fine. Not super precise, but good enough and easy.
Also, add a loop so the benchmark is like 1 second long, I'm old enough to be worried about anything smaller than 1/18 seconds.
Magma7404•9mo ago
DenisM•9mo ago
gus_massa•9mo ago
Also, add a loop so the benchmark is like 1 second long, I'm old enough to be worried about anything smaller than 1/18 seconds.