Select Rows with a certain value present

user_14576516 0
Viewed 113

I have created a correlation matrix using the Pandas method df.corr(). I need to select the rows in which I have even one value greater than 0.7. How do I achieve this?

P.S.: There are over 60 columns in my dataset, hence I need to vectorize this. Also it is important to understand that there is a value 1.0 in every row (every column is perfectly correlated with itself). Hence I only need to choose rows which have values greater than 0.7 apart from that 1.0.

pandas numpy python
3 Answers

Try Boolean masking and the loc accessor:

# Since the maximum value is 1, this should also work:
df.loc[((df>0.7) & (df<1)).any(1)]
# OR
df.loc[df.replace(1, float('NaN')).gt(0.7).any(1)]
user_142898920

Use replace to avoid matching all rows with 1, and compare for greater than 0.7 if at least one row:

df[df.replace(1,0).gt(0.7).any(axis=1)]

compare

user_29010020

If I understand correctly, just use gt:

print(df[df.gt(0.7).any(axis=1)])
user_87083640
Related