Calculating mean and standard deviation and ignoring 0 values

user_13038912 0
Viewed 525

I have a list of lists with sublists all of which contain float values. For example the one below has 2 lists with sublists each:

 mylist =  [[[2.67, 2.67, 0.0, 0.0], [2.67, 2.67, 2.0, 2.0], [2.67, 2.67, 2.0, 2.0], [2.67, 2.67, 2.0, 2.0], [2.67, 2.67, 2.0, 2.0], [2.67, 2.67, 2.0, 2.0], [0.0, 2.67, 2.0, 2.0], [2.67, 2.67, 2.0, 2.0], [2.67, 2.67, 2.0, 2.0], [2.67, 2.67, 2.0, 2.0]], [[2.67, 2.67, 2.0, 2.0], [0.0, 2.67, 2.0, 2.0], [2.67, 2.67, 2.0, 2.0], [2.67, 2.67, 2.0, 2.0], [2.67, 2.67, 2.0, 2.0], [0.0, 0.0, 0.0, 0.0], [2.67, 2.67, 2.0, 2.0], [2.67, 2.67, 2.0, 2.0], [2.67, 2.67, 2.0, 2.0], [2.67, 2.67, 2.0, 2.0], [2.67, 2.67, 2.0, 2.0], [2.67, 2.67, 2.0, 2.0], [2.67, 2.67, 2.0, 2.0], [2.67, 2.67, 2.0, 2.0]]]

I want to calculate the standard deviation and the mean of the sublists and what I applied was this:

mean = [statistics.mean(d) for d in mylist]
stdev = [statistics.stdev(d) for d in mylist]

but it takes also the 0.0 values that I do not want because I turned them to 0 in order not to be empty ones. Is there a way to ignore these 0s as they do not exist in the sublist?To not take them under consideration at all? I could not find a way for how I am doing it.

sublist mean standard-deviation list python
4 Answers

You can use numpy's nanmean and nanstd functions.

import numpy as np


def zero_to_nan(d):
    array = np.array(d)
    array[array == 0] = np.NaN
    return array


mean = [np.nanmean(zero_to_nan(d)) for d in mylist]
stdev = [np.nanstd(zero_to_nan(d)) for d in mylist]
user_76485780

You can do this with a list comprehension.

The following lambda function flattens the nested list into a single list and filters out all zeros:

flatten = lambda nested: [x for sublist in nested for x in sublist if x != 0]

Note that the list comprehension has two for and one ifstatement similar to this code snippet, which does essentially the same:

flat_list = []

for sublist in nested:
   for x in sublist:
       if x != 0:
           flat_list.append(x)

To apply this to your list you can use map. The map function will return an iterator. To get a list we need to pass the iterator to list:

flat_list = list(map(flatten, myList))

Now you can calculate the mean and standard deviation:

mean = [statistics.mean(d) for d in flat]
stdev = [statistics.stdev(d) for d in flat]

print(mean)
print(stdev)
user_74544550 
mean = [statistics.mean(d) for d in mylist if d != 0]
stdev = [statistics.stdev(d) for d in mylist if d != 0]
user_135523260

Try:

mean = [statistics.mean([k for k in d if k]) for d in mylist]
stdev = [statistics.stdev([k for k in d if k]) for d in mylist]
user_129097760
Related