Applying dataframe-returning function to every row of base dataframe

Question

Toy example

Suppose that base_df is the tiny dataframe shown below:

In [221]: base_df
Out[221]: 
     seed
I S      
0 a     0
  b     1
1 a     2
  b     3

Note that base_df has a 2-level multi-index for the rows. (Part of the problem here involves "propagating" this multi-index's values in a derived dataframe.)

Now, the function fn (definition given at the end of this post) takes an integer seed as argument and returns a 1-column dataframe indexed by string keys¹. For example:

In [222]: fn(0)
Out[222]: 
              F
key            
01011  0.592845
10100  0.844266

In [223]: fn(1)
Out[223]: 
              F
key            
11110  0.997185
01000  0.932557
11100  0.128124

I want to generate a new dataframe, in essence, by applying fn to every row of base_df, and concatenating the resulting dataframes vertically. More specifically, the desired result would look like this:

                  F
I S key            
0 a 01011  0.592845
    10100  0.844266
  b 11110  0.997185
    01000  0.932557
    11100  0.128124
1 a 01101  0.185082
    01110  0.931541
  b 00100  0.070725
    11011  0.839949
    11111  0.121329
    11000  0.569311

IOW, conceptually, the desired dataframe is obtained by generating one "sub-dataframe" for each row of base_df, and concatenating these sub-dataframes vertically. The sub-dataframe corresponding to each row has a 3-level multi-index. The first two levels (I and S) of this multi-index come from base_df's multi-index value for that row, while its last level (key), as well as the values for the (lone) F column come from the dataframe returned by fn for that row's seed value.

The part I'm not clear on is how to propagate the row's original multi-index value to the rows of the dataframe created by fn for that row's seed value.

IMPORTANT: I'm looking for a way to do this that is agnostic to the names of the base_df's multi-index's levels, and their number.

I tried the following

base_df.apply(lambda row: fn(row.seed), axis=1)

...but the evaluation fails with the error

ValueError: Shape of passed values is (4, 2), indices imply (4, 1)

Is there some convenient way to do what I'm trying to do?

Here's the definition of fn. Its internals are unimportant as far as this question is concerned. What matters is that it takes an integer seed as argument, and returns a dataframe, as described earlier.

import numpy
def fn(seed, _spec='{{0:0{0:d}b}}'.format(5)):
    numpy.random.seed(int(seed))
    n = numpy.random.randint(2, 5)
    r = numpy.random.rand(n)
    k = map(_spec.format, numpy.random.randint(0, 31, size=n))
    result = pandas.DataFrame(r, columns=['F'], index=k)
    result.index.name = 'key'
    return result

^{¹ In this example, these keys happen to correspond to the binary representation of some integer between 0 and 31, inclusive, but this fact plays no role in the question.}

Applying dataframe-returning function to every row of base dataframe

1 Answers