I will focus at the algorithm level only here.
The problem is equivalent to calculate the probability than after K steps, the sum of the values is greater than N.
Let ut call p the probability that at a given step, on element is selected
p = 1/(M+1)
The idea is to represent the probability set of one trial in a polynomial form:
A[x] = p * (1 + x^2 + x^3 + ... + x^{M-1})
Then after K steps, the probability set is given by:
F[x] = p^K * (1 + x^2 + x^3 + ... + x^{M-1})^K = p^K A[x]^K
At the end, if F[x] = sum_i p_i x^i, then the probability is given by:
P = sum_{i >= N} p_i
The problem is then to calculate the polynomial power A[x]^k.
A first attempt consits in
calculating it recursively: then we obtained something equivalent to be solution provided in the first answer, same complexity.
A second attempt consist in implicitely considering the binary representation of K.
Pseudo-code:
F[x] = 1
T[x] = A[x]
Power = K
while (Power != 0)
if (Power mod 2 == 1) F[x] = F[x] * T[x]
T[x] = T[x] * T[x]
Power = Power / 2
end while
The number of steps is equel to log2(K).
At each iteration, the degree of the polynomial is multiplied by two: the complexity is dominated by the last step,
where the order of the polynomial is equal to KM.
This last step has a complexity O(K^2 M^2).
If C = K^2 M^2, The global complexity of the power calculation is O(C + C/4 + C/8 + ...) = O(C) = O(K^2M^2).
This method doesn't seem to provide any advantage with respect to the first one (I may have made an error in the complexity evaluation!).
A third method consists in considering that a polynomial multiplication is equivalent to a convolution, and can be performed via a FFT process. As the final size is O(KM), then the complexity is O(M K (log M + log K)).
Describing this method in details here would take much time. You could find many references
on internet on this subject, for example here
Side note: a pity not being able to insert latex math formula here ...