merge a list of dicts by a certain key

You can use itertools.groupby:

>>> from itertools import groupby
>>> result = []
>>> for key, group in groupby(sorted(sample, key=lambda x:x['a']), key=lambda x:x.pop('a')):
        result.append({'a':key, 'd':[*group]})
>>> result
[{'a': 1, 'd': [{'b': 2, 'c': 3}, {'b': 2, 'c': 4}]},
 {'a': 2, 'd': [{'b': 2, 'c': 5}, {'b': 3, 'c': 5}]}]

NOTE: You don't need sorted if it is guaranteed that the list of dicts come sorted by the value of key a.

Combine by key:

dict_list = [{'a':1, 'b':2, 'c':3}, {'a':1, 'b':2, 'c':4}, {'a':2, 'b':2, 'c':5}, {'a':2, 'b':3, 'c':5}]
new_dict = {}

for d in dict_list:
    a = d.pop('a', None)
    if new_dict.get(a):
         new_dict[a].append(d)
    else:
        new_dict[a] = [d]

Convert to list:

final_list = [{'a': key, 'd': value} for key, value in new_dict.items()]
print(final_list)

[{'a': 1, 'd': [{'c': 3, 'b': 2}, {'c': 4, 'b': 2}]}, {'a': 2, 'd': [{'c': 5, 'b': 2}, {'c': 5, 'b': 3}]}]

sample = [{'a':1, 'b':2, 'c':3}, {'a':1, 'b':2, 'c':4}, {'a':2, 'b':2, 'c':5}, {'a':2, 'b':3, 'c':5}]


tmp = {}
for v in sample:
    tmp.setdefault(v['a'], []).append(v)
    del v['a']

out = [{'a': k, 'd': v} for k, v in tmp.items()]

from pprint import pprint
pprint(out)

Prints:

[{'a': 1, 'd': [{'b': 2, 'c': 3}, {'b': 2, 'c': 4}]},
 {'a': 2, 'd': [{'b': 2, 'c': 5}, {'b': 3, 'c': 5}]}]

This may be a bit twisty code unfortunately, but it works:

from itertools import groupby

sample = [{'a':1, 'b':2, 'c':3},
          {'a':1, 'b':2, 'c':4},
          {'a':2, 'b':2, 'c':5},
          {'a':2, 'b':3, 'c':5}]

main_key = "a"

print(
    [{main_key:k,
      "d": [{kk: vv for kk, vv in dct.items() if kk != main_key}
            for dct in v]}
     for k, v in groupby(sample, lambda d:d[main_key])]
)

gives:

[{'a': 1, 'd': [{'b': 2, 'c': 3}, {'b': 2, 'c': 4}]},
 {'a': 2, 'd': [{'b': 2, 'c': 5}, {'b': 3, 'c': 5}]}]

(output slightly pretty-printed for readability)

An alternative solution using Pandas for your query.

import pandas as pd
sample = [{'a':1, 'b':2, 'c':3}, {'a':1, 'b':2, 'c':4}, {'a':2, 'b':2, 'c':5}, {'a':2, 'b':3, 'c':5}]

df=pd.DataFrame(sample)

This would create a DataFrame df using the above sample list. The next step would be to iterate through the GroupBy Object and create the output as required.

final_list=[]
for i, temp_df in df.groupby('a'):
    temp_list=[]
    for j in temp_df.index:
        temp_list.append({'b':temp_df.loc[:,'b'][j],'c':temp_df.loc[:,'c'][j]})
    final_list.append({'a':temp_df.loc[:,'a'][j],'d':temp_list})