How can I map and reduce my list of dictionaries with Python

user_9447403 0
Viewed 735

I have this list of dictionaries:

[{'topic_id': 1, 'average': 5.0, 'count': 1}, {'topic_id': 1, 'average': 8.0, 'count': 1}, {'topic_id': 2, 'average': 5.0, 'count': 1}]

I would like to map and reduce (or group) to have a result like this:

[
    {
        'topic_id': 1,
        'count': 2,
        'variance': 3.0,
        'global_average': 6.5
        },
    {
        'topic_id': 2,
        'count': 1,
        'variance': 5.0,
        'global_average': 5.0
    }
]

Something that calculate the variance (max average - min average) and sum the count of items too.

What I have already did:

Before I just tried sum the count changing the structure of the dictionary, and making the key be the topic_id and value the count, my result was:

result = sorted(dict(functools.reduce(operator.add, map(collections.Counter, data))).items(), reverse=True)

this was just the first try.

mapreduce python-3.x list dictionary python
4 Answers

Here is an attempt using itertools.groupby to group the data based on the topic_id:

import itertools

data = [{'topic_id': 1, 'average': 5.0, 'count': 1}, {'topic_id': 1, 'average': 8.0, 'count': 1}, {'topic_id': 2, 'average': 5.0, 'count': 1}]

# groupby
grouper = itertools.groupby(data, key=lambda x: x['topic_id'])

# holder for output
output = []

# iterate over grouper to calculate things
for key, group in grouper:

    # variables for calculations
    count = 0
    maxi = -1
    mini = float('inf')
    total = 0

    # one pass over each dictionary
    for g in group:
        avg = g['average']
        maxi = avg if avg > maxi else maxi
        mini = avg if avg < mini else mini
        total += avg
        count += 1

    # write to output
    output.append({'total_id':key,
                   'count':count,
                   'variance':maxi-mini,
                   'global_average':total/count})

Giving this output:

[{'total_id': 1, 'count': 2, 'variance': 3.0, 'global_average': 6.5},
 {'total_id': 2, 'count': 1, 'variance': 0.0, 'global_average': 5.0}]

Note that the 'variance' for the second group is 0.0 here instead of 5.0; this is different from your expected output, but I would guess this is what you want?

user_133869790

You could achieve this with some comprehensions, a map, and the mean function from the built-in statistics module.

from statistics import mean
data = [
    {
        'topic_id': 1, 
        'average': 5.0, 
        'count': 1
    }, {
        'topic_id': 1, 
        'average': 8.0, 
        'count': 1
    }, {
        'topic_id': 2, 
        'average': 5.0, 
        'count': 1
    }
]
# a set of unique topic_id's
keys = set(i['topic_id'] for i in data)
# a list of list of averages for each topic_id
averages = [[i['average'] for i in data if i['topic_id'] == j] for j in keys]
# a map of tuples of (counts, variances, averages) for each topic_id
stats = map(lambda x: (len(x), max(x) - min(x), mean(x)), averages)
# finally reconstruct it back into a list
result = [
    {
        'topic_id': key, 
        'count': count, 
        'variance': variance, 
        'global_average': average
    } for key, (count, variance, average) in zip(keys, stats)
]
print(result)

Returns

[{'topic_id': 1, 'count': 2, 'variance': 3.0, 'global_average': 6.5}, {'topic_id': 2, 'count': 1, 'variance': 0.0, 'global_average': 5.0}]
user_124796390

If you are willing to use pandas, this seems like an appropriate use case:

import pandas as pd

data = [{'topic_id': 1, 'average': 5.0, 'count': 1}, {'topic_id': 1, 'average': 8.0, 'count': 1}, {'topic_id': 2, 'average': 5.0, 'count': 1}]

# move to dataframe
df = pd.DataFrame(data)

# groupby and get all desired metrics
grouped = df.groupby('topic_id')['average'].describe()
grouped['variance'] = grouped['max'] - grouped['min']

# rename columns and remove unneeded ones
grouped = grouped.reset_index().loc[:, ['topic_id', 'count', 'mean', 'variance']].rename({'mean':'global_average'}, axis=1)

# back to list of dicts
output = grouped.to_dict('records')

output is:

[{'topic_id': 1, 'count': 2.0, 'global_average': 6.5, 'variance': 3.0},
 {'topic_id': 2, 'count': 1.0, 'global_average': 5.0, 'variance': 0.0}]
user_133869790

You can also try to use the agg functionality of pandas dataframe like this

import pandas as pd

f = pd.DataFrame(d).set_index('topic_id')

def var(x):
    return x.max() - x.min()

out = f.groupby(level=0).agg(count=('count', 'sum'),
        global_average=('average', 'mean'),
        variance=('average', var))
user_68663930
Related