@StefanPochmann said in How is this tested?:

Can you please share how you calculated that? Thanks.

The short answer is Chebyshev's inequality.

The long answer is

Let X_i be the indicator function of the event that at the ith time we draw a number from 0, 1, 2, ..., 9 and we get number 0. Then if your random generator is correct, Prob(X_i = 1) = 0.1, Prob(X_i = 0) = 0.9. EX_i = 0.1 Var(X_i) = 0.1 * (1 - 0.1)^2 + 0.9 * (0 - 0.1)^2 = 0.09.

If we draw 1M times, then the random variable "# of times we get 0" is just X_1 + X_2 + ... +X_1M.

Let \bar{X}_n = (X_1 + X_2 + ... + X_n) / n. E(\bar{X}_n) = E(X_i) = 0.1. Var(\bar{X_n}) = Var(X_i) / n. So here Var(\bar{X_1M}) = 0.09/1M.

Lastly apply Chebyshev's inequality:

P(we get 0.12) = P(X_1 + ...X_n = 0.12M) = P(\bar{X}_n = 0.12) < P( |\bar{X} - 0.1 | >= 0.02) <= 0.09/(0.0004 * 1M) = 0.0225%.

The last number is way smaller than 0.1%, plus the strict inequality and the fact that Chebyshev is not quite tight. So my estimate was pretty conservative. Note this is only a necessary condition, which means your generator has to satisfies this but satisfying that would not be sufficient. For example you could return the i % 10 for the i th call.