Combining p values using scipy

user_3291993 0
Viewed 899

I have to combine p values and get one p value. I'm using scipy.stats.combine_pvalues function, but it is giving very small combined p value, is it normal?

e.g.:

>>> import scipy
>>> p_values_list=[8.017444955844044e-06, 0.1067379119652372, 5.306374345615846e-05, 0.7234201655194492, 0.13050605094545614, 0.0066989543716175, 0.9541246420333787]
>>> test_statistic, combined_p_value = scipy.stats.combine_pvalues(p_values_list, method='fisher',weights=None)
>>> combined_p_value
4.331727536209026e-08

As you see, combined_p_value is smaller than any given p value in the p_values_list? How can it be?

Thanks in advance, Burcak

p-value scipy statistics python
2 Answers

It is correct, because you are testing all of your p-values come from a random uniform distribution. The alternate hypothesis is that at least one of them is true. Which in your case is very possible.

We can simulate this, by drawing from a random uniform distribution 1000 times, the length of your p-values:

import numpy as np
from scipy.stats import combine_pvalues
from matplotlib import pyplot as plt

random_p = np.random.uniform(0,1,(1000,len(p_values_list)))
res = np.array([combine_pvalues(i,method='fisher',weights=None) for i in random_p])
plt.hist(fisher_p)

enter image description here

From your results, the chi-square is 62.456 which is really huge and no where near the simulated chi-square above.

One thing to note is that the combining you did here does not take into account directionality, if that is possible in your test, you might want to consider using stouffer's Z along with weights. Also another sane way to check is to run simulation like the above, to generate list of p-values under the null hypothesis and see how they differ from what you observed.

Interesting paper but maybe a bit on the statistics side

user_122584590

I am by no means an expert in this field, but am interested in your question. Following some reading of wiki it seems to me that the combined_p_value tells you the likelihood of all p-values in the list been obtained under the same null-hypothesis. Which is very unlikely considering two extremely small values. Your set has two extremely small values: 1st and 3rd. If the thought process I described is correct, removing any of them should yield a much higher p-value, which is indeed the case:
remove 1st: p-value of 0.00010569305282803985
remove 3rd: p-value of 2.4713196031837724e-05
In conclusion, I think that this is a correct way of interpreting the meta-analysis that combine_pvalues actually describes.

user_90462470
Related