Fishers Exact Test from Pandas Dataframe

user_3050230 0
Viewed 1361

I'm trying to work out the best way to create a p-value using Fisher's Exact test from four columns in a dataframe. I have already extracted the four parts of a contingency table, with 'a' being top-left, 'b' being top-right, 'c' being bottom-left and 'd' being bottom-right. I have started including additional calculated columns via simple pandas calculations, but these aren't necessary if there's an easier way to just use the 4 initial columns. I have over 1 million rows when including an additional set (x.type = high), so want to use an efficient method. So far this is my code:

import pandas as pd
import glob
import math
path = r'directory_path'
all_files = glob.glob(path + "/*.csv")
li = []

for filename in all_files:
    df = pd.read_csv(filename, index_col=None, header=0)
    li.append(df)

frame = pd.concat(li, axis=0, ignore_index=True)
frame['a+b'] = frame['a'] + frame['b']
frame['c+d'] = frame['c'] + frame['d']
frame['a+c'] = frame['a'] + frame['c']
frame['b+d'] = frame['b'] + frame['d']

As an example of this data, 'frame' currently shows:

    ID(n)   a   b   c   d   i   x.name  x.type  a+b     c+d     a+c     b+d
0   1258065     5   28  31  1690    1754    Albumin     low     33  1721    36  1718
1   1132105     4   19  32  1699    1754    Albumin     low     23  1731    36  1718
2   898621  4   30  32  1688    1754    Albumin     low     34  1720    36  1718
3   573158  4   30  32  1688    1754    Albumin     low     34  1720    36  1718
4   572975  4   23  32  1695    1754    Albumin     low     27  1727    36  1718
...     ...     ...     ...     ...     ...     ...     ...     ...     ...     ...     ...     ...
666646  12435   1   0   27  1726    1754    WHR     low     1   1753    28  1726
666647  15119   1   0   27  1726    1754    WHR     low     1   1753    28  1726
666648  17053   1   2   27  1724    1754    WHR     low     3   1751    28  1726
666649  24765   1   3   27  1723    1754    WHR     low     4   1750    28  1726
666650  8733    1   1   27  1725    1754    WHR     low     2   1752    28  1726

Is the best way to convert these to a numpy array and process it through iteration, or keep it in pandas? I assume that I can't use math functions within a dataframe (I've tried math.comb(), which didn't work in a dataframe). I've also tried using pyranges for its fisher method but it seems it doesn't work with my environment (python 3.8).

Any help would be much appreciated!

pandas combinations statistics python-3.x python
1 Answers

Following the answer here which came from the author of pyranges (i think), let's say you data is something like:

import pandas as pd 
import scipy.stats as stats
import numpy as np

np.random.seed(111)
df = pd.DataFrame(np.random.randint(1,100,(1000000,4)))
df.columns=['a','b','c','d']
df['ID'] = range(1000000)

df.head()

    a   b   c   d   ID
0   85  85  85  87  0
1   20  42  67  83  1
2   41  72  58  8   2
3   13  11  66  89  3
4   29  15  35  22  4

You convert it into a numpy array and did it like in the post:

c = df[['a','b','c','d']].to_numpy(dtype='uint64')

from fisher import pvalue_npy

_, _, twosided = pvalue_npy(c[:, 0], c[:, 1], c[:, 2], c[:, 3])
df['odds'] = (c[:, 0] * c[:, 3]) / (c[:, 1] * c[:, 2])

df['pvalue'] = twosided

Or you can fit it directly:

_, _, twosided = pvalue_npy(df['a'].to_numpy(np.uint), df['b'].to_numpy(np.uint), 
                            df['c'].to_numpy(np.uint), df['d'].to_numpy(np.uint))

df['odds'] = (df['a'] * df['d']) / (df['b'] * df['c'])
df['pvalue'] = twosided
user_122584590
Related