Ask HN: Are your code benchmarks statistically valid?
1•DenisM•7h ago
It appears that some benchmarking tools provide average or percentile values but no information about statistical validity of the result. What did you observe or do in your career?
Comments
Magma7404•7h ago
I haven't got a clue since I have never used any benchmark tool in my whole life. Could you tell us in which professional context you would use that kind of tool?
DenisM•6h ago
Run a benchmark to compare performance of my product before and after making a change in code.
gus_massa•44m ago
I run it 5 times and drop the two extremes. If the middle tree don't overlap, I'm probably fine. Not super precise, but good enough and easy.
Also, add a loop so the benchmark is like 1 second long, I'm old enough to be worried about anything smaller than 1/18 seconds.
Magma7404•7h ago
DenisM•6h ago
gus_massa•44m ago
Also, add a loop so the benchmark is like 1 second long, I'm old enough to be worried about anything smaller than 1/18 seconds.