Find duplicate values in two arrays, Python

user_8510149 0
Viewed 5439

I have two arrays (A and B) with about 50 000 values in each. Every value represents an ID. I want to create a pandas dataframe with three columns, col1: values from array A, col2: values from array B, col3: a string with the labels "unique" or "duplicate". In each array the ID:s are unique.

The arrays is of different length. So I can't do something like this to get started.

a = np.array([1, 2, 3, 4, 5])
a = np.array([5, 6, 7, 8, 9, 10])
pd.DataFrame({'a':a, 'a':b})

I was then thinking to create a different pandas dataframe, also with three columns. One for ID, another for which array the ID comes from (a or b). And thereafter group on ID and count occurrences. if >=2 then we have a duplicate.

But I couldn’t figure out how get to numpy arrays after one another in the same column (like rbind in R) and at the same time create the other column based on which array the value come from.

Most likely there are far better solutions that those I have suggested above. Any ideas?

pandas numpy python
3 Answers

For finding duplicate elements in two arrays, use numpy.intersect1d:

In [458]: a = np.array([1, 2, 3, 4, 5])

In [459]: b = np.array([5, 6, 7, 8, 9, 10])

In [462]: np.intersect1d(a,b)
Out[462]: array([5])
user_58208140

Convert the array into series and then concat them to create dataframe

a = np.array([1, 2, 3, 4, 5,])
b = np.array([5, 6, 7, 8, 9, 10])

s1 = pd.Series(a, name = 'a')
s2 = pd.Series(b, name = 'b')
pd.concat([s1, s2], axis = 1)

     a  b
0   1.0 5
1   2.0 6
2   3.0 7
3   4.0 8
4   5.0 9
5   NaN 10
user_150916060

Try with merge + indicator

out = pd.DataFrame({'a':a}).merge(pd.DataFrame({'b':b}), left_on='a',right_on='b',indicator=True,how='outer')
Out[210]: 
     a     b      _merge
0  1.0   NaN   left_only
1  2.0   NaN   left_only
2  3.0   NaN   left_only
3  4.0   NaN   left_only
4  5.0   5.0        both
5  NaN   6.0  right_only
6  NaN   7.0  right_only
7  NaN   8.0  right_only
8  NaN   9.0  right_only
9  NaN  10.0  right_only
user_79645270
Related