Multiply the values for each row in two different dataframes

Question

I'm building genetic algorithm to feature selection in python. I have extracted features from my datas, then I divided into two dataframe, 'train' and 'test' dataframe. How can I multiple the values for each row in 'population' dataframe (each individu) and 'train' dataframe?

'train' dataframe:

   feature0   feature1   feature2   feature3   feature4   feature5
0  18.279579  -3.921346  13.611829  -7.250185 -11.773605 -18.265003   
1  17.899545 -15.503942  -0.741729  -0.053619  -6.734652   4.398419   
4  16.432750 -22.490190  -4.611659 -15.247781 -13.941488  -2.433374   
5  15.905368  -4.812785  18.291712   3.742221   3.631887  -1.074326   
6  16.991823 -15.946251   8.299577   8.057511   8.057510  -1.482333

'population' dataframe:

      0     1     2     3     4     5     
0     1     1     0     0     0     1     
1     0     1     0     1     0     0     
2     0     0     0     0     0     1     
3     0     0     1     0     1     1

Multiplying each row in 'population' to all rows in 'train'. It will results:

1) From population row 1:

   feature0   feature1   feature2   feature3   feature4   feature5
0  18.279579  -3.921346          0          0          0 -18.265003   
1  17.899545 -15.503942          0          0          0   4.398419   
4  16.432750 -22.490190          0          0          0  -2.433374   
5  15.905368  -4.812785          0          0          0  -1.074326   
6  16.991823 -15.946251          0          0          0  -1.482333

2) From population row 2:

   feature0   feature1   feature2   feature3   feature4   feature5
0          0  -3.921346          0  -7.250185          0          0
1          0 -15.503942          0  -0.053619          0          0   
4          0 -22.490190          0 -15.247781          0          0   
5          0  -4.812785          0   3.742221          0          0   
6          0 -15.946251          0   8.057511          0          0

And so on...

user_2901002 · Answer

If need loop (slow if large data):

for i, x in population.iterrows():
    print (train * x.values)

    feature0   feature1  feature2  feature3  feature4   feature5
0  18.279579  -3.921346       0.0      -0.0      -0.0 -18.265003
1  17.899545 -15.503942      -0.0      -0.0      -0.0   4.398419
4  16.432750 -22.490190      -0.0      -0.0      -0.0  -2.433374
5  15.905368  -4.812785       0.0       0.0       0.0  -1.074326
6  16.991823 -15.946251       0.0       0.0       0.0  -1.482333
   feature0   feature1  feature2   feature3  feature4  feature5
0       0.0  -3.921346       0.0  -7.250185      -0.0      -0.0
1       0.0 -15.503942      -0.0  -0.053619      -0.0       0.0
4       0.0 -22.490190      -0.0 -15.247781      -0.0      -0.0
5       0.0  -4.812785       0.0   3.742221       0.0      -0.0
6       0.0 -15.946251       0.0   8.057511       0.0      -0.0
   feature0  feature1  feature2  feature3  feature4   feature5
0       0.0      -0.0       0.0      -0.0      -0.0 -18.265003
1       0.0      -0.0      -0.0      -0.0      -0.0   4.398419
4       0.0      -0.0      -0.0      -0.0      -0.0  -2.433374
5       0.0      -0.0       0.0       0.0       0.0  -1.074326
6       0.0      -0.0       0.0       0.0       0.0  -1.482333
   feature0  feature1   feature2  feature3   feature4   feature5
0       0.0      -0.0  13.611829      -0.0 -11.773605 -18.265003
1       0.0      -0.0  -0.741729      -0.0  -6.734652   4.398419
4       0.0      -0.0  -4.611659      -0.0 -13.941488  -2.433374
5       0.0      -0.0  18.291712       0.0   3.631887  -1.074326
6       0.0      -0.0   8.299577       0.0   8.057510  -1.482333

Or each row separately:

print (train * population.values[0])

    feature0   feature1  feature2  feature3  feature4   feature5
0  18.279579  -3.921346       0.0      -0.0      -0.0 -18.265003
1  17.899545 -15.503942      -0.0      -0.0      -0.0   4.398419
4  16.432750 -22.490190      -0.0      -0.0      -0.0  -2.433374
5  15.905368  -4.812785       0.0       0.0       0.0  -1.074326
6  16.991823 -15.946251       0.0       0.0       0.0  -1.482333

Or for MultiIndex DataFrame:

d = pd.concat([train * population.values[i] for i in range(population.shape[0])],
               keys=population.index.tolist())
print (d)

      feature0   feature1   feature2   feature3   feature4   feature5
0 0  18.279579  -3.921346   0.000000  -0.000000  -0.000000 -18.265003
  1  17.899545 -15.503942  -0.000000  -0.000000  -0.000000   4.398419
  4  16.432750 -22.490190  -0.000000  -0.000000  -0.000000  -2.433374
  5  15.905368  -4.812785   0.000000   0.000000   0.000000  -1.074326
  6  16.991823 -15.946251   0.000000   0.000000   0.000000  -1.482333
1 0   0.000000  -3.921346   0.000000  -7.250185  -0.000000  -0.000000
  1   0.000000 -15.503942  -0.000000  -0.053619  -0.000000   0.000000
  4   0.000000 -22.490190  -0.000000 -15.247781  -0.000000  -0.000000
  5   0.000000  -4.812785   0.000000   3.742221   0.000000  -0.000000
  6   0.000000 -15.946251   0.000000   8.057511   0.000000  -0.000000
2 0   0.000000  -0.000000   0.000000  -0.000000  -0.000000 -18.265003
  1   0.000000  -0.000000  -0.000000  -0.000000  -0.000000   4.398419
  4   0.000000  -0.000000  -0.000000  -0.000000  -0.000000  -2.433374
  5   0.000000  -0.000000   0.000000   0.000000   0.000000  -1.074326
  6   0.000000  -0.000000   0.000000   0.000000   0.000000  -1.482333
3 0   0.000000  -0.000000  13.611829  -0.000000 -11.773605 -18.265003
  1   0.000000  -0.000000  -0.741729  -0.000000  -6.734652   4.398419
  4   0.000000  -0.000000  -4.611659  -0.000000 -13.941488  -2.433374
  5   0.000000  -0.000000  18.291712   0.000000   3.631887  -1.074326
  6   0.000000  -0.000000   8.299577   0.000000   8.057510  -1.482333

And select by xs:

print (d.xs(0))

    feature0   feature1  feature2  feature3  feature4   feature5
0  18.279579  -3.921346       0.0      -0.0      -0.0 -18.265003
1  17.899545 -15.503942      -0.0      -0.0      -0.0   4.398419
4  16.432750 -22.490190      -0.0      -0.0      -0.0  -2.433374
5  15.905368  -4.812785       0.0       0.0       0.0  -1.074326
6  16.991823 -15.946251       0.0       0.0       0.0  -1.482333

user_2336654 · Answer

I'd use numpy broadcasting to do it all in one go...

train_ = pd.DataFrame(
    (train.values * pop.values[:, None]).reshape(-1, train.shape[1]),
    pd.MultiIndex.from_product([pop.index, train.index]),
    train.columns
)

train_

      feature0   feature1   feature2   feature3   feature4   feature5
0 0  18.279579  -3.921346   0.000000  -0.000000  -0.000000 -18.265003
  1  17.899545 -15.503942  -0.000000  -0.000000  -0.000000   4.398419
  4  16.432750 -22.490190  -0.000000  -0.000000  -0.000000  -2.433374
  5  15.905368  -4.812785   0.000000   0.000000   0.000000  -1.074326
  6  16.991823 -15.946251   0.000000   0.000000   0.000000  -1.482333
1 0   0.000000  -3.921346   0.000000  -7.250185  -0.000000  -0.000000
  1   0.000000 -15.503942  -0.000000  -0.053619  -0.000000   0.000000
  4   0.000000 -22.490190  -0.000000 -15.247781  -0.000000  -0.000000
  5   0.000000  -4.812785   0.000000   3.742221   0.000000  -0.000000
  6   0.000000 -15.946251   0.000000   8.057511   0.000000  -0.000000
2 0   0.000000  -0.000000   0.000000  -0.000000  -0.000000 -18.265003
  1   0.000000  -0.000000  -0.000000  -0.000000  -0.000000   4.398419
  4   0.000000  -0.000000  -0.000000  -0.000000  -0.000000  -2.433374
  5   0.000000  -0.000000   0.000000   0.000000   0.000000  -1.074326
  6   0.000000  -0.000000   0.000000   0.000000   0.000000  -1.482333
3 0   0.000000  -0.000000  13.611829  -0.000000 -11.773605 -18.265003
  1   0.000000  -0.000000  -0.741729  -0.000000  -6.734652   4.398419
  4   0.000000  -0.000000  -4.611659  -0.000000 -13.941488  -2.433374
  5   0.000000  -0.000000  18.291712   0.000000   3.631887  -1.074326
  6   0.000000  -0.000000   8.299577   0.000000   8.057510  -1.482333

You can access just the one corresponding to the ith row or population with train_.loc[i]

train_.loc[3]

   feature0  feature1   feature2  feature3   feature4   feature5
0       0.0      -0.0  13.611829      -0.0 -11.773605 -18.265003
1       0.0      -0.0  -0.741729      -0.0  -6.734652   4.398419
4       0.0      -0.0  -4.611659      -0.0 -13.941488  -2.433374
5       0.0      -0.0  18.291712       0.0   3.631887  -1.074326
6       0.0      -0.0   8.299577       0.0   8.057510  -1.482333

ROUGH TIME TEST
I'm too lazy to do more robust testing

%%timeit
pd.DataFrame(
    (train.values * pop.values[:, None]).reshape(-1, train.shape[1]),
    pd.MultiIndex.from_product([pop.index, train.index]),
    train.columns
)

%%timeit
res = pop.iloc[np.repeat(np.arange(len(pop)), len(train))]
res = res.set_index(np.tile(train.index, len(pop)), append=True).add_prefix('feature')
res.mul(train, level=1)

%%timeit
pd.concat([train * pop.values[i] for i in range(pop.shape[0])],
               keys=pop.index.tolist())

571 µs ± 10.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
1.42 ms ± 18 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
1.7 ms ± 69.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

user_1240268 · Answer

Once you set the columns of population to match train you can use *:

In [11]: population.columns = train.columns

In [12]: train * population.iloc[0]
Out[12]:
    feature0   feature1  feature2  feature3  feature4   feature5
0  18.279579  -3.921346       0.0      -0.0      -0.0 -18.265003
1  17.899545 -15.503942      -0.0      -0.0      -0.0   4.398419
4  16.432750 -22.490190      -0.0      -0.0      -0.0  -2.433374
5  15.905368  -4.812785       0.0       0.0       0.0  -1.074326
6  16.991823 -15.946251       0.0       0.0       0.0  -1.482333

You can make a MultiIndex (as recommended by @jezrael) very efficiently using np.tile and np.repeat:

In [11]: res = population.iloc[np.repeat(np.arange(len(population)), len(train))]

In [12]: res = res.set_index(np.tile(train.index, len(population)), append=True)

In [13]: res
Out[13]:
     feature0  feature1  feature2  feature3  feature4  feature5
0 0         1         1         0         0         0         1
  1         1         1         0         0         0         1
  4         1         1         0         0         0         1
  5         1         1         0         0         0         1
  6         1         1         0         0         0         1
1 0         0         1         0         1         0         0
  1         0         1         0         1         0         0
  4         0         1         0         1         0         0
  5         0         1         0         1         0         0
  6         0         1         0         1         0         0
2 0         0         0         0         0         0         1
  1         0         0         0         0         0         1
  4         0         0         0         0         0         1
  5         0         0         0         0         0         1
  6         0         0         0         0         0         1
3 0         0         0         1         0         1         1
  1         0         0         1         0         1         1
  4         0         0         1         0         1         1
  5         0         0         1         0         1         1
  6         0         0         1         0         1         1

In [14]: res.mul(train, level=1)
Out[14]:
      feature0   feature1   feature2   feature3   feature4   feature5
0 0  18.279579  -3.921346   0.000000  -0.000000  -0.000000 -18.265003
  1  17.899545 -15.503942  -0.000000  -0.000000  -0.000000   4.398419
  4  16.432750 -22.490190  -0.000000  -0.000000  -0.000000  -2.433374
  5  15.905368  -4.812785   0.000000   0.000000   0.000000  -1.074326
  6  16.991823 -15.946251   0.000000   0.000000   0.000000  -1.482333
1 0   0.000000  -3.921346   0.000000  -7.250185  -0.000000  -0.000000
  1   0.000000 -15.503942  -0.000000  -0.053619  -0.000000   0.000000
  4   0.000000 -22.490190  -0.000000 -15.247781  -0.000000  -0.000000
  5   0.000000  -4.812785   0.000000   3.742221   0.000000  -0.000000
  6   0.000000 -15.946251   0.000000   8.057511   0.000000  -0.000000
2 0   0.000000  -0.000000   0.000000  -0.000000  -0.000000 -18.265003
  1   0.000000  -0.000000  -0.000000  -0.000000  -0.000000   4.398419
  4   0.000000  -0.000000  -0.000000  -0.000000  -0.000000  -2.433374
  5   0.000000  -0.000000   0.000000   0.000000   0.000000  -1.074326
  6   0.000000  -0.000000   0.000000   0.000000   0.000000  -1.482333
3 0   0.000000  -0.000000  13.611829  -0.000000 -11.773605 -18.265003
  1   0.000000  -0.000000  -0.741729  -0.000000  -6.734652   4.398419
  4   0.000000  -0.000000  -4.611659  -0.000000 -13.941488  -2.433374
  5   0.000000  -0.000000  18.291712   0.000000   3.631887  -1.074326
  6   0.000000  -0.000000   8.299577   0.000000   8.057510  -1.482333

Multiply the values for each row in two different dataframes

3 Answers