Out of context, the expression "the uncertainty in the number of trials" would refer to missing knowledge in terms of how many trials actually ran.

In the context of the post this doesn't make sense, so the reader is left to hypothesize what the writer actually meant.

I think "uncertainty due to the number of trials" would be clearer.

I guess I struggle to understand why under estimation is worse than over estimation, when the final result is a ranking. It seems like they're equally likely to produce an incorrect ranking!

The problems are really underspecified for statisticians, too. Leetcode is normally very clear on the requirements.

> it is very important that the uncertainty in the number of trials is taken into account because over-estimating a fraction is a costly mistake.

This is not some precise jargon that is meaningless to the layman but completely clearly specified to a professional statistician. It's more like the specification written by your non-technical product manager for how some technical feature should work. A skilled data scientist will have the experience and the context to figure out what it's probably asking for, but he might write down a few more clarifying details before giving it to a junior on his team to implement.

If testing these kind of guess-what-the-stakeholder probably-means skills is the point of this test, it's quite good at it. But that's not what leetcode is for.

Put more simply: suppose I have a coin that might be biased. I decide to asses this via repeatedly flipping it—if it’s biased, it will disproportionately land on heads or tails. If I flip 10 times and get 4 heads/6 tails, I don’t have the power to make a confident assessment of any bias. On the other hand, if I flip it 100 times and get 40 heads/60 tails, I am a bit more certain. At 1000 flips, with 400 heads/600 tails, I am extremely confident the coin is biased. Even though the fraction of heads is identically 40% across all three sets of flips, the underlying counts yield very different amounts of confidence on how close to 40% the coin’s bias is. The first question is a way of rigorously quantifying this confidence.

I don’t think this is “leetcode for statisticians.” This question (and the other two) are all examples of concrete, real-world problems that people across a variety of quantitative disciplines frequently encounter.

In fact, the first question is directly relevant to voting on this site. When sorting replies by fraction of upvotes, how should the forum software rank a new reply with 1 upvote/0 downvotes, versus an older reply with 4 upvotes/1 downvote? What about an older, more controversial reply with 20 upvotes/7 downvotes? 15 upvotes/2 downvotes?

No worries, I wasn't really trying to explain it anyways, as much as seeking confirmation from the rest of HN that this question is ill specified. Judging from responses, yes, it is.

If "binomial distribution" and "confidence interval" are unfamiliar terms then you probably are not prepared to pass OP's "statistical reasoning test" regardless. I think most engineers wouldn't, and I only understood the intent of question 1 because my pandemic lockdown project was reading a stats textbook cover to cover.

You mean (n+1)/(n+m+2) and yes, this is Laplace's rule of succession. It won't give you a confidence interval, but it gives you a posterior point estimate.

If you want a rough 95 % confidence interval without complicated maths, the Agresti–Coull interval is useful. It's computed as if the distribution was normal, but pretending there were two more successes and failures instead.

Bingo.

The answer looks a bit simplistic compared to the question (as I interpreted it at least). In order to estimate the risk of incorrect ordering you have to calculate P(p2 > p1) where p1 and p2 are drawn from different beta distributions. AFAIK there’s no closed form expression for that probability (so Monte Carlo is one possible approach).