Loop Produces Incorrect stats and P Value

user_2812625 0
Viewed 32

I am trying to streamline me doing t tests and found a method to loop through each column based on a filtering flag. however when I do the loop function the results are

  1. All the same Stats and P Value
  2. Different if I were to do it as a stand alone.

1: Wrong

for col in df1.filter(like='DELTA').columns:
stats.mannwhitneyu(df1[df1['COHORT_FLAG']==1]['MH_IP_DELTA'] ,df1[df1['COHORT_FLAG']==0]['MH_IP_DELTA'])
print(col + ': ' + 'Stats=%.3f, p=%.3f' % (stat, p))

2: Right:

stats.mannwhitneyu(df1[df1['COHORT_FLAG']==1]['MH_IP_DELTA'] ,df1[df1['COHORT_FLAG']==0]['MH_IP_DELTA'])

Any idea why the loop doesnt work correctly and is there another way to loop through a large record set? I dont want to manually input [col] everytime.

pandas scipy statistics python
1 Answers

There's no dataframe to play with included in your question, but I can point to where the issue may be coming from. First, I'll format your code, so that we're all on the same page with what we're looking at:

1 ("Right"):

for col in df1.filter(like='DELTA').columns:
    stats.mannwhitneyu(df1[df1['COHORT_FLAG']==1]['MH_IP_DELTA'] ,df1[df1['COHORT_FLAG']==0]['MH_IP_DELTA'])
    print(col + ': ' + 'Stats=%.3f, p=%.3f' % (stat, p))

2 ("Wrong"):

stats.mannwhitneyu(df1[df1['COHORT_FLAG']==1]['MH_IP_DELTA'] ,df1[df1['COHORT_FLAG']==0]['MH_IP_DELTA'])

The first thing to notice is that there's no difference between the key lines in both the "right" and the "wrong" code snippets. This tells us that the issue is with the way the loop is constructed. Looking at the loop, we see that col never shows up except in the print() statement. The fact that the loop produces all the same values confirms that the loop is not looping over different columns.

Without a dataframe, I don't know what columns you're interested in, and I'm not overly familiar with the stats function you're using. But you need to tell your program what column is of interest. Perhaps you change 'MH_IP_DELTA' to col if this is the column that shold be changing. Not sure if this is the variable of interest, cause I can't see what's going on in your dataframe, but that would be my guess.

Which column it is, and where you need to change it is up to you to figure out, but the issue is that your code, as written, simply runs the same piece of code X number of times where X is the number of columns.

user_91500120
Related