Apply function to pandas row-row cross product

user_11486611 0
Viewed 947

I have two pandas DataFrames / Series containing one row each.

df1 = pd.DataFrame([1, 2, 3, 4])
df2 = pd.DataFrame(['one', 'two', 'three', 'four'])

I now want to get all possible combinations into an n*n matrix / DataFrame with values for all cross-products being the output from a custom function.

def my_function(x, y):
    return f"{x}:{y}"

This should therefore result in:

df = pd.DataFrame([['1:one', '2:one', '3:one', '4:one'],
                   ['1:two', '2:two', '3:two', '4:two'],
                   ['1:three', '2:three', '3:three', '4:three'],
                   ['1:four', '2:four', '3:four', '4:four']])

         0        1        2        3
0    1:one    2:one    3:one    4:one
1    1:two    2:two    3:two    4:two
2  1:three  2:three  3:three  4:three
3   1:four   2:four   3:four   4:four

While I can build my own matrix through itertools.product, this seems like a very inefficient way for larger datasets and I was wondering if there is a more pythonic way. Thank you in advance.

pandas dataframe python
5 Answers

You also can use pd.DataFrame constructor with apply:

pd.DataFrame(index=df2.squeeze(), columns=df1.squeeze()).apply(lambda x: x.name.astype(str)+':'+x.index)

Output:

            1        2        3        4                                        
one      1:one    2:one    3:one    4:one
two      1:two    2:two    3:two    4:two
three  1:three  2:three  3:three  4:three
four    1:four   2:four   3:four   4:four

Explanation:

First, with pd.DataFrame constructor, first build and empty dataframe with index and columns defined from df2 and df1 respectively. Using pd.DataFrame.squeeze, we convert those single column dataframes into a pd.Series.

Next, using pd.DataFrame.apply, we can apply a lambda function which adds the strings from the column name with a colon and the dataframe index for each column of the dataframe.

This yeilds a new dataframe with indexing and desired values.

user_63615310

Let us try np.add.outer

df = pd.DataFrame(np.add.outer(df1[0].astype(str).values,':'+df2[0].values).T)
Out[258]: 
         0        1        2        3
0    1:one    2:one    3:one    4:one
1    1:two    2:two    3:two    4:two
2  1:three  2:three  3:three  4:three
3   1:four   2:four   3:four   4:four
user_79645270

Another way using np.tile:

pd.DataFrame(np.tile(df1[0][:,None],df2.shape[0])).astype(str).add(":"+df2[0]).T

Or similar but without transposing courtesy @Ch3ster

pd.DataFrame(np.repeat(df1[0].astype(str)[None,:],df2.shape[0],axis=0)).add(':'+df2[0])

         0        1        2        3
0    1:one    2:one    3:one    4:one
1    1:two    2:two    3:two    4:two
2  1:three  2:three  3:three  4:three
3   1:four   2:four   3:four   4:four

EDIT,

For using alongside your function you can also use a cross join:

def my_function(x, y):
    return f"{x}:{y}"

u = df1.assign(k=1).merge(df2.assign(k=1),on='k').drop('k',1).to_numpy()
arr = (np.array([*map(lambda x: my_function(*x),u)])
         .reshape((df1.shape[0],df2.shape[0]),order='F'))

print(arr,"\n---------------------------------------------------\n",pd.DataFrame(arr))

[['1:one' '2:one' '3:one' '4:one']
 ['1:two' '2:two' '3:two' '4:two']
 ['1:three' '2:three' '3:three' '4:three']
 ['1:four' '2:four' '3:four' '4:four']] 
---------------------------------------------------
         0        1        2        3
0    1:one    2:one    3:one    4:one
1    1:two    2:two    3:two    4:two
2  1:three  2:three  3:three  4:three
3   1:four   2:four   3:four   4:four
user_98406370

You can add them but flatten the 1st df using numpy.ndarray.ravel

pd.DataFrame(df1.astype(str).to_numpy().ravel() + ':' + df2.to_numpy())

         0        1        2        3
0    1:one    2:one    3:one    4:one
1    1:two    2:two    3:two    4:two
2  1:three  2:three  3:three  4:three
3   1:four   2:four   3:four   4:four
user_124164530

Just for completeness, the above answers work for simple use cases. For more complex custom functions, this is probably the easiest (although somewhat ugliest) option:

df = []
for i in df1.iterrows():
    row = [] 
    for j in df2.iterrows():
        row.append(my_function(i[1][0], j[1][0]))
    df.append(row)

pd.DataFrame(df)
user_114866110
Related