Derive multiple df from single df such that each df has no NaN values

Viewed 60

I want to convert this table

0   thg   John     3.0
1   thg  James     4.0
2   mol    NaN     5.0
3   mol    NaN     NaN
4   lob    NaN     NaN

In this following tables

df1
movie   name  rating
0   thg   John     3.0
1   thg  James     4.0

df2
    movie  rating
2   mol     5.0

df3
    movie
3   mol  
4   lob  

Where each dataframe has no Nan value, Also tell method if I need to separate with respect to blank value instead of Nan.

2 Answers

I think that start of a new target DataFrame should occur not only when the number of NaN values changes (compared to previous row), but also when this number is the same, but NaN values are in different columns.

So I propose the following formula:

dfs = [g.dropna(how='all',axis=1) for _,g in
    df.groupby(df.isna().ne(df.isna().shift()).any(axis=1).cumsum())]

You can print partial DataFrames (any number of them) running:

n = 0
for grp in dfs:
    print(f'\ndf No {n}:\n{grp}')
    n += 1

The advantage of my solution over the other becomes obvious when you add to the source DataFrame another row containing:

5   NaN    NaN    3.0

It contains also 1 non-null value (like two previous rows). The other solution will treat all these rows as one partial DataFrame containing:

  movie  rating
3   mol     NaN
4   lob     NaN
5   NaN     3.0

as you can see, with NaN values, whereas my solution divides these rows into 2 separate DataFrames, without any NaN.

create a list of dfs , with a groupby and dropna:

dfs = [g.dropna(how='all',axis=1) for _,g in df.groupby(df.isna().sum(1))]
print(dfs[0],'\n\n',dfs[1],'\n\n',dfs[2])

Or dict:

d = {f"df{e+1}": g[1].dropna(how='all',axis=1) 
       for e,g in enumerate(df.groupby(df.isna().sum(1)))}
print(d['df1'],'\n\n',d['df2'],'\n\n',d['df3']) #read the keys of d

  movie   name  rating
0   thg   John     3.0
1   thg  James     4.0 

   movie  rating
2   mol     5.0 

   movie
3   mol
4   lob
Related