How to count values since the first non nan value?

user_13039962 0
Viewed 168

I have this df:

         CODE    TMAX
0        000130  NaN
1        000130  NaN
2        000130  32.0
3        000130  32.2
4        000130  31.1
5        158328  22.5
6        158328  8.8
7        158328  NaN
8        158328  NaN
9        158328  9.2
...      ...     ...

I want to count the number of non nan values and the number of nan values in the 'TMAX' column. But i want to count since the first non NaN value.

Expected result: 6 non nan values and 2 NaN values.

How can i do this?

Thanks in advance.

pandas python
3 Answers

Use Series.notna with Series.cummax for filter out first NaNs and then count by Series.value_counts with rename index values by dict:

m = df.TMAX.notna()
s = m[m.cummax()].value_counts().rename({True:'non NaNs',False:'NaNs'})
print (s)
non NaNs    6
NaNs        2
Name: TMAX, dtype: int64
user_29010020 
result = (df.TMAX
            .truncate(before=df.TMAX.first_valid_index())
            .mask(pd.Series.notna, other="non-NaN")
            .value_counts(dropna=False))

where we truncate the series before its first valid index i.e., first non-Nan index, and then turn it into a NaN or not NaN series with mask. Then we count the values (including NaNs),

to get

>>> result

non-NaN    6
NaN        2
user_93321870

Use first_valid_index to find the first non-NaN index and filter. Then use isna to create a boolean mask and count the values.

output = (
    df.loc[df.TMAX.first_valid_index():, 'TMAX']
    .isna()
    .value_counts()
    .rename({True: 'NaN', False: 'notNaN'})
)

Output

notNaN    6
NaN       2
Name: TMAX, dtype: int64
user_51200490
Related