Smoothing Categorical Output

user_9877412 0
Viewed 254

I have a list of outputs obtained from a cow behavior detection model. Even in a video when a cow is laying, often time it identifies as standing and vice versa. In each video frame, a classification result is given by the model and we are appending it into a list. Let's assume after 20 frames, we have a series of output as follows -

behavious_cow_1 = ["stand","stand","stand","stand","lying", "stand","stand", "eating", "stand","stand","stand","stand","lying""stand","stand","stand","stand","stand","stand","lying"]

Out of 20 classification results, we have 4 misclassification; 3 lyings, and 1 eating. However, the whole time the cow was sitting at a place. If the list only contained numerical values like - 1,2,3..., I would have opted for moving average to change the misclassification. Is there any Scipy, Pandas, Numpy function that can smooth the categorical output? I am thinking about taking previous 3 and next 3 values to determine the current category.

pandas scipy math numpy python
2 Answers

I used the following solution -

import scipy.stats
window_length = 7
behave = ["stand","stand","stand","stand","lying","lying", "eating"]
most_freq_val = lambda x: scipy.stats.mode(x)[0][0]
smoothed = [most_freq_val(behave[i:i+window_length]) for i in range(0,len(behave)-window_length+1)]

I tried the solution posted by Hugolmn but it broke at a point. In the rolling mode, the window width is provided by the user (7 here). In a certain width, if more than one values are present in the same number of times, the code does not work. It's more like - you tried to find the statistical mode (most common item) of a list but it got more than one item with the same highest frequency.

user_98774120

I am myself very surprised that a function such as mode() does not work with a rolling window in pandas. However, I still found a decent solution to your problem

First, create a pandas Series with categorical datatype:

df = pd.Series(sample, dtype='category')

Now you can see that df.cat.categories returns the list of categories in your data, and df.cat.codes the codes associated to them. We can use the latter to apply a rolling mode with a width of 7 (3 previous, the value, and the next 3):

df.cat.codes
0     3
1     3
2     3
3     3
4     1
5     3
6     3
7     0
8     3
9     3
10    3
11    3
12    2
13    3
14    3
15    3
16    3
17    3
18    1
dtype: int8

df.cat.codes.rolling(7, center=True, min_periods=0).apply(lambda x: x.mode())
0     3.0
1     3.0
2     3.0
3     3.0
4     3.0
5     3.0
6     3.0
7     3.0
8     3.0
9     3.0
10    3.0
11    3.0
12    3.0
13    3.0
14    3.0
15    3.0
16    3.0
17    3.0
18    3.0
dtype: float64

Finally, you can map the codes to get the strings back:

(df.cat.codes
  .rolling(7, center=True, min_periods=0)
  .apply(lambda x: x.mode())
  .map(dict(enumerate(df.cat.categories)))
)
0     stand
1     stand
2     stand
3     stand
4     stand
5     stand
6     stand
7     stand
8     stand
9     stand
10    stand
11    stand
12    stand
13    stand
14    stand
15    stand
16    stand
17    stand
18    stand
dtype: object

And there you go ! You recovered your strings after applying a rolling mode on their codes !

user_137650850
Related