How to split a series by the longest repetition of a number in python?

Viewed 66
df = pd.DataFrame({
    'label':[f"subj_{i}" for i in range(28)],
    'data':[i for i in range(1, 14)] + [1,0,0,0,2] + [0,0,0,0,0,0,0,0,0,0]
})

I have a dataset something like that. It looks like:

enter image description here

I want to cut it at where the longest repetitions of 0s occur, so I want to cut at index 18, but I want to leave index 14-16 intact. So far I've tried stuff like:

  • Counters
cad_recorder = 0
new_index = []
for i,row in tqdm(temp_df.iterrows()):
    if row['cadence'] == 0:
        cad_recorder += 1
        new_index.append(i)
* But obviously that won't work since the indices will be rewritten at each occurrance of zero. 
  • I also tried a dictionary, but I'm not sure how to compare previous and next values using iterrows.
  • I also took the rolling mean for X rows at a time, and if its zero then I got an index. But then I got stuck at actually inferring the range of indices. Or finding the longest sequence of zeroes.

Edit: A friend of mine suggested the following logic, which gave the same results as @shubham-sharma. The poster's solution is much more pythonic and elegant.

def find_longest_zeroes(df):
    '''
    Finds the index at which the longest reptitions of <1 values begin
    '''
    current_length = 0
    max_length = 0
    start_idx = 0
    max_idx = 0


    for i in range(len(df['data'])):
        if df.iloc[i,9] <= 1:
            if current_length == 0:
                start_idx = i
            current_length += 1

            if current_length > max_length:
                max_length = current_length
                max_idx = start_idx
        else:
            current_length = 0
    return max_idx

The code I went with following @shubham-sharma's solution:

cut_us_sof = {}
og_df_sof = pd.DataFrame()
cut_df_sof = pd.DataFrame()

for lab in df['label'].unique():
    temp_df = df[df['label'] == lab].reset_index(drop=True)
    mask = temp_df['data'] <= 1 # some values in actual dataset were 0.0000001
    counts = temp_df[mask].groupby((~mask).cumsum()).transform('count')['data']
    idx = counts.idxmax()
    # my dataset's trailing zeroes are usually after 200th index. But I also didn't want to remove trailing zeroes < 500 in length
    if (idx > 2000) & (counts.loc[idx] > 500):
        cut_us_sof[lab] = idx
        og_df_sof = og_df_sof.append(temp_df)
        cut_df_sof = cut_df_sof.append(temp_df.iloc[:idx,:])
1 Answers

We can use boolean masking and cumsum to identify the blocks of zeros, then groupby and transform these blocks using count followed by idxmax to get the starting index of the block having the maximum consecutive zeros

m = df['data'].eq(0)
idx = m[m].groupby((~m).cumsum()).transform('count').idxmax()

print(idx)

18
Related