Append missing category into rows

user_11629229 0
Viewed 57

i have a set of id having some category. However, i want each id have the same number of category which can be specified as df.id.category.unique().

For example: Input

df1 = {"id": [1,1,1,2,2,3,3,3,3],
      "category": ["a","b","e","a","d","a","b","c","d"]
      }

output1 = pd.DataFrame(df1)
output1
Out[57]: 
   id category
0   1        a
1   1        b
2   1        e
3   2        a
4   2        d
5   3        a
6   3        b
7   3        c
8   3        d

The output should be: Output

df2 = {"id": [1,1,1,1,1,2,2,2,2,2,3,3,3,3,3],
      "category": sum([["a","b","c","d","e"] for _ in range(3)], [])}

output2 = pd.DataFrame(df2)
output2
Out[58]: 
    id category
0    1        a
1    1        b
2    1        c
3    1        d
4    1        e
5    2        a
6    2        b
7    2        c
8    2        d
9    2        e
10   3        a
11   3        b
12   3        c
13   3        d
14   3        e

If possible, I hope to have fast optimization. Thanks a lot!

pandas numpy algorithm python
3 Answers

Use itertools.product:

from  itertools import product

df = pd.DataFrame(product(output1['id'].unique(), output1['category'].unique()),
                  columns=['id','category'])
    
print (df)
    id category
0    1        a
1    1        b
2    1        e
3    1        d
4    1        c
5    2        a
6    2        b
7    2        e
8    2        d
9    2        c
10   3        a
11   3        b
12   3        e
13   3        d
14   3        c

Or MultiIndex.from_product with MultiIndex.to_frame:

df = (pd.MultiIndex.from_product([output1['id'].unique(), output1['category'].unique()], 
                   names=['id','category'])
        .to_frame(index=False))
                     
print (df)
    id category
0    1        a
1    1        b
2    1        e
3    1        d
4    1        c
5    2        a
6    2        b
7    2        e
8    2        d
9    2        c
10   3        a
11   3        b
12   3        e
13   3        d
14   3        c
user_29010020

You might use numpy.tile together with numpy.repeat for this as follows

import numpy as np
id_col = np.repeat([1,2,3,4,5],5).reshape(-1,1)
category_col = np.tile(["a","b","c","d","e"],5).reshape(-1,1)
arr = np.hstack([id_col,category_col])
print(arr)

output

[['1' 'a']
 ['1' 'b']
 ['1' 'c']
 ['1' 'd']
 ['1' 'e']
 ['2' 'a']
 ['2' 'b']
 ['2' 'c']
 ['2' 'd']
 ['2' 'e']
 ['3' 'a']
 ['3' 'b']
 ['3' 'c']
 ['3' 'd']
 ['3' 'e']
 ['4' 'a']
 ['4' 'b']
 ['4' 'c']
 ['4' 'd']
 ['4' 'e']
 ['5' 'a']
 ['5' 'b']
 ['5' 'c']
 ['5' 'd']
 ['5' 'e']]
user_107859750

One option is the complete function from pyjanitor, to expose the missing rows:

#pip install git+https://github.com/pyjanitor-devs/pyjanitor.git
import pandas as pd
import janitor as jn

output1.complete('id', 'category')
Out[1280]: 
    id category
0    1        a
1    1        b
2    1        e
3    2        a
4    2        d
5    3        a
6    3        b
7    3        c
8    3        d
9    1        c
10   1        d
11   2        b
12   2        c
13   2        e
14   3        e
user_71757130
Related