Pandas - how to find a sequence of 6 zero in any row in a data frame, and replace it to NaN?

Viewed 117

How can I find every row in a data frame that contains 6 zero (in adjacent columns), and then to replace this 6 zero to Nan?

1 Answers

We could torture Numpy functions to get us some clever way to do this. But it is simpler and likely more efficient to loop with Numba.

import numpy as np
import pandas as pd
from numba import njit

Setup

np.random.seed([3, 14])
df = pd.DataFrame(
    np.random.randint(2, size=(100_000, 20)),
    columns=[*'ABCDEFGHIJKLMNOPQRST']
)

Numba

@njit
def replace_too_many_zeros(a, k):
    i = [0][:0]
    j = [0][:0]
    for i_, row in enumerate(a):
        count_zeros = 0
        for j_, col in enumerate(row):
            if col == 0:
                count_zeros += 1
            else:
                if count_zeros >= k:
                    j += list(range(j_ - count_zeros, j_))
                    i += [i_] * count_zeros
                count_zeros = 0
    return np.array(i), np.array(j)

Numpy slice assignment

a = df.to_numpy()
i, j = replace_too_many_zeros(a, 6)
a[i, j] = np.nan
df.loc[:] = a
Related