Does pandas fillna have column and row number limitations?

Question

I use pandas fillna in a dataframe of 4253532 rows × 151 columns, it doesn't work. The return is the df with no change. I tried different method, all leave the df unchanged.
Same code works well in a 3741814 rows × 4 columns dataframe.

df2=df.fillna(method='ffill',axis=1)

df.fillna(value=10,inplace=True)

I tried return a new df and inplace, both don't work.
I am sure each row has at least one valid value.
Anyone meet the same problem? Does pandas have a limitation?

More information \

df:

print(b.to_markdown())

|         |       8000001 |   8000016 |   8000017 |   8000018 |   8000019 |
|--------:|--------------:|----------:|----------:|----------:|----------:|
| 4253527 |   2.00711e+07 |       nan |       nan |       nan |       nan |
| 4253528 | nan           |       nan |       nan |       nan |       nan |
| 4253529 | nan           |       nan |       nan |       nan |       nan |
| 4253530 | nan           |       nan |       nan |       nan |       nan |
| 4253531 | nan           |       nan |       nan |       nan |       nan |

data type:

b.dtypes

ITEM_OBJECT_ID
8000001    float64
8000016    float64
8000017    float64
8000018    float64
8000019    float64
dtype: object

is nan:

print(np.isnan(b).to_markdown())

|         |   8000001 |   8000016 |   8000017 |   8000018 |   8000019 |
|--------:|----------:|----------:|----------:|----------:|----------:|
| 4253527 |         0 |         1 |         1 |         1 |         1 |
| 4253528 |         1 |         1 |         1 |         1 |         1 |
| 4253529 |         1 |         1 |         1 |         1 |         1 |
| 4253530 |         1 |         1 |         1 |         1 |         1 |
| 4253531 |         1 |         1 |         1 |         1 |         1 |

fillna:

print(b.fillna(100).to_markdown())

|         |       8000001 |   8000016 |   8000017 |   8000018 |   8000019 |
|--------:|--------------:|----------:|----------:|----------:|----------:|
| 4253527 |   2.00711e+07 |       nan |       nan |       nan |       nan |
| 4253528 | nan           |       nan |       nan |       nan |       nan |
| 4253529 | nan           |       nan |       nan |       nan |       nan |
| 4253530 | nan           |       nan |       nan |       nan |       nan |
| 4253531 | nan           |       nan |       nan |       nan |       nan |

print(b.fillna(method='ffill',axis=1).to_markdown())

|         |       8000001 |   8000016 |   8000017 |   8000018 |   8000019 |
|--------:|--------------:|----------:|----------:|----------:|----------:|
| 4253527 |   2.00711e+07 |       nan |       nan |       nan |       nan |
| 4253528 | nan           |       nan |       nan |       nan |       nan |
| 4253529 | nan           |       nan |       nan |       nan |       nan |
| 4253530 | nan           |       nan |       nan |       nan |       nan |
| 4253531 | nan           |       nan |       nan |       nan |       nan |

user_6936489 · Answer

There are no limits to pandas as far as I know. I have used the fillna method on dataframes of many millions rows without any trouble.

For example :

import pandas as pd
import random
import numpy as np



df = pd.DataFrame(
        {
                'A' : [random.choice([np.nan] + list(range(10))) for x in range(int(1e6))],
                'B' : [random.choice([np.nan] + list(range(10))) for x in range(int(1e6))],
                'C' : [random.choice(['a', 'b', 'c'])  for x in range(int(1e6))]
        }
)
df = df.fillna("bla")
print(
      df[(df.values=="bla").any(axis=1)]
      )

It works as well with 1e7 rows. So maybe, this is a trouble in your first dataframe ?

Does pandas fillna have column and row number limitations?

1 Answers