Statistics for data in multiple tuples of names and data

user_6852339 0
Viewed 173

A beginner here looking for some help with getting statistics from a list of tuples, where a tuple contains names and the data.

I have a list of the following format:

list = [(name1, name2, name3, name(...), value1, value2, value3, value(...))]

Example:

mylist = [('red', 1, 100, 0.075, 0.055, 1.88),
      ('red', 1, 100, 0.0825, 0.05, 1.84),
      ('red', 1, 2, 3.7, 0.08, 4.20),
      ('green', 2, 2, 0.37, 0.8, 0.9),
      ('green', 2, 2, 0.85, 0.35, 1.24)]

What I need is means and standard deviations of values from tuples where the names are identical.

The output should be:

output = [(name1, name2, name3, name(...), value1_mean, value1_stdev, value2_mean, value2_stdev, value3_mean, value3_stdev, value(...)_mean, value(...)_stdev)]

For all unique name1, name2, name3, name(...) combinations.

In the case of the list above the desired output is the following:

output = [('green', 2, 2, 0.61, 0.33941125496954283, 0.575, 0.3181980515339464, 1.07, 0.2404163056034261),
      ('red', 1, 2, 3.7, 0, 0.08, 0, 4.2, 0),
      ('red', 1, 100, 0.07875, 0.005303300858899111, 0.052500000000000005, 0.003535533905932736, 1.8599999999999999, 0.02828427124746177)]

I managed to get this to work, not in an elegant way, where I am limited to 3 names and 3 values:

import statistics

mylist = [('red', 1, 100, 0.075, 0.055, 1.88),
          ('red', 1, 100, 0.0825, 0.05, 1.84),
          ('red', 1, 2, 3.7, 0.08, 4.20),
          ('green', 2, 2, 0.37, 0.8, 0.9),
          ('green', 2, 2, 0.85, 0.35, 1.24)]

d_0 = []
d_1 = []
d_2 = []

for i in mylist:

    d_0.append(i[0])
    d_1.append(i[1])
    d_2.append(i[2])

s_d_0 = set(d_0)
s_d_1 = set(d_1)
s_d_2 = set(d_2)

for d0 in s_d_0:
    for d1 in s_d_1:
        for d2 in s_d_2:
            for c in [3,4,5]:
                exec('v' + str(c) + '_' + str(d0) + str(d1) + str(d2) + ' = []')

results = []
for t in mylist:
    for d0 in s_d_0:
        for d1 in s_d_1:
            for d2 in s_d_2:
                if d0 == t[0] and d1 == t[1] and d2 == t[2]:
                    for c in [3,4,5]:
                        exec('v' + str(c) + '_' + str(d0) + str(d1) + str(d2) + '.append( t[' + str(c) + '])')


for d0 in s_d_0:
    for d1 in s_d_1:
        for d2 in s_d_2:
            asd = [d0, d1, d2]
            for c in [3, 4, 5]:
                length = 0
                exec('length = len(v' + str(c) + '_' + str(d0) + str(d1) + str(d2) + ')')
                if length == 0:
                    exec('mean' + str(c) + '_' + str(d0) + str(d1) + str(d2) + ' = 0')
                    exec('stdev' + str(c) + '_' + str(d0) + str(d1) + str(d2) + ' = 0')
                if not length == 0:

                    exec('mean' + str(c) + '_' + str(d0) + str(d1) + str(d2) + ' = statistics.mean(v' + str(c) + '_' + str(d0) + str(d1) + str(d2) + ')')
                    exec('stdev' + str(c) + '_' + str(d0) + str(d1) + str(d2) + ' = 0')

                if not length == 1 and not length == 0:
                    exec('stdev' + str(c) + '_' + str(d0) + str(d1) + str(d2) + ' = statistics.stdev(v' + str(
                        c) + '_' + str(d0) + str(d1) + str(d2) + ')')

                exec('fgh = (mean' + str(c) + '_' + str(d0) + str(d1) + str(d2) + ', stdev' + str(c) + '_' + str(d0) + str(d1) + str(d2) + ')')
                asd.append(fgh)
            results.append(asd)

final = []
for z in results:
    if z[3][0] is not 0:
        final.append(z)

output = []
for fin in final:

    final3 = []
    for fiin in fin:
        if not isinstance(fiin, tuple):

            final3.append(fiin)
        if isinstance(fiin, tuple):

            fiin1, fiin2 = fiin[0], fiin[1]

            final3.append(fiin1)
            final3.append(fiin2)

    output.append(tuple(final3))

for f in output:
    print(f)

Is there a way of obtaining the same output in a more robust way, perhaps using numpy or pandas? Preferably where I could specify a number different than 3 for how many of the left values of the tuple define names.

Thanks!

pandas tuples statistics python-3.x numpy
1 Answers

I'm not sure I get you question very precisely. The following function takes your myList as argument and returns 

output = [(name1, name2, name3, name(...), value1_mean, value1_stdev, value2_mean, value2_stdev, value3_mean, value3_stdev, value(...)_mean, value(...)_stdev)]

import numpy as np

mylist = [('red', 1, 100, 0.075, 0.055, 1.88),
      ('red', 1, 100, 0.0825, 0.05, 1.84),
      ('red', 1, 2, 3.7, 0.08, 4.20),
      ('green', 2, 2, 0.37, 0.8, 0.9),
      ('green', 2, 2, 0.85, 0.35, 1.24)]

def getStats(myList):
    # dict containing mean and std for each name, ie dico[name1][value_number]=[list of value_numbers]
    dico={}

    # Fill dico
    for t in myList: # For each tuple
        name=t[0] # Get the name
        if name not in dico: # Check wether the name is already registered in dico. If not, add it
            dico[name]={nv:[] for nv in range(len(t)-1)}

        l=[]
        for nval in range(len(t)-1): # Browse values
            dico[name][nval].append(t[nval+1]) # Store them at the right place in dico

    # Create the output
    output=[]

    # Fill output
    for name in dico: # Browse through dico's names
        lval=[name]
        for nval in dico[name]: # For each value_i from tuple
            l=dico[name][nval]
            mu=np.mean(l) # Compute mean...
            std=np.std(l) # ... and standard deviation thanks to NumPy
            lval.append(mu)
            lval.append(std)
        output.append(tuple(lval)) # Format output to match your desires

    return(output)

>>> getStats(mylist)
[('red', 1.0, 0.0, 67.33333333333333, 46.197643037521104, 1.2858333333333334, 1.7070763668395805, 0.06166666666666667, 0.013123346456686353, 2.64, 1.1032074449833391), ('green', 2.0, 0.0, 2.0, 0.0, 0.61, 0.24, 0.575, 0.22500000000000003, 1.07, 0.16999999999999998)]

Hope it will fit your needs ! I recommend you to dive into dict usage [https://www.w3schools.com/python/python_dictionaries.asp] to learn more about this powerful tool, especially when you have to deal with multi-dimensional dataset.

user_106141340
Related