Combine Python List of Dict's optionally based on properties lists values

user_9179498 0
Viewed 74

I am struggling to come up with some code that does all of the following, wondering if anyone with a bit more knowledge of python can help :)!

It should:

  • Removes duplicates from an inputted list - combining the 'account' property into a 'accounts' list - but only if all the other properties are the same.
  • If anything except from 'account' differs - have it in its own property of the list.

Perhaps better as example input and output:

input = [
    {
      "name":"core",
      "account":"foo-bar",
      "options": {
          "foo": "bar"
      },
    },
    {
      "name":"core",
      "account":"zed-bar",
      "options": {
          "foo": "bar"
      },
    },
    {
      "name":"core",
      "account":"tup-bar",
      "options": {
          "foo": "zed"
      },
    },
]

Note: How the 3rd properties 'options>foo' is different from the other two.

output = [
    {
      "name":"core",
      "accounts":[
          "foo-bar",
          "zed-bar"
      ],
      "options": {
          "foo": "bar"
      },
    },
    {
      "name":"core",
      "accounts":[
          "tup-bar"
      ],
      "options": {
          "foo": "zed"
      },
    },
]

Notice how it combines the first two elements -- as name and options are the same. But does not combine the 3rd element as its options are different.

Thanks for any help with this, I have been rattling my brain for the past two days trying various combinations.

python
4 Answers

Based on your input!

i = 0
j = 0
while i<len(data):
    d1 = data[i]
    while j<len(data):
        if j>i:
            d2 = data[j]
            if d1["name"] == d2["name"] and d1["options"] == d2["options"]:
                if d1["account"]:
                    d1["account"] = [d1["account"]]
                else:
                    d1["account"] = []
                if d2["account"]:
                    d1["account"].append(d2["account"])
                del data[j]
        j+=1
    i +=1

Output:

[{'name': 'core',
  'account': ['foo-bar', 'zed-bar'],
  'options': {'foo': 'bar'}},
 {'name': 'core', 'account': 'tup-bar', 'options': {'foo': 'zed'}}]
user_119465050

Here is my attempt using pandas' duplicated and assert_series_equal:

import pandas as pd
from pandas.testing import assert_series_equal

input1 = [
    {
      "name":"core",
      "account":"foo-bar",
      "options": {
          "foo": "bar"
      },
    },
    {
      "name":"core",
      "account":"zed-bar",
      "options": {
          "foo": "bar"
      },
    },
    {
      "name":"core",
      "account":"tup-bar",
      "options": {
          "foo": "zed"
      },
    },
]

df_input1 = pd.DataFrame(input1)
df_input1.options = df_input1.options.astype(str)

var_to_append = 'account'
vars_to_check = ['name','options']

idx_duplicated = df_input1.duplicated(subset=vars_to_check)

df_new = df_input1[~idx_duplicated]
df_dup = df_input1[idx_duplicated]

df_new[var_to_append] = df_new[var_to_append].apply(lambda x: [x])

for i in df_dup.index:
    row_dup = df_dup.loc[i]
    for j in df_new.index:
        row_new = df_new.loc[j]
        try:        
            assert_series_equal(row_dup[vars_to_check], 
                                row_new[vars_to_check],
                            check_names=False)
            df_new.loc[j, var_to_append] += [row_dup[var_to_append]]
            print('added', row_dup[var_to_append], 'to', row_new[var_to_append])
        except:
            pass

[df_new.to_dict(orient='records')]
user_49758360

Using two list to keep reference by same index.

values = []
account = []

for i in input_list:
    key = (i['name'], i['options'])
    if key in values:
        index = values.index(key)
        account[index].append(i['account'])
    else:
        values.append(key)
        account.append([i['account']])

result = [{'name':name, 'accounts':accounts, 'options':options}
    for (name, options), accounts in zip(values, account)]
user_119361350

Given your input, you could try something like this:

# Merge entries with the same options.
merged = [{
    "name": d['name'],
    "accounts": list(d2['account'] for d2 in input if d['options'] == d2['options']),
    "options": d['options'],
} for d in input]

# Remove duplicate entries from the output.
output = []
for d in merged:
    if d not in output:
        # If the list contains one element, then replace the list with it.
        if len(d['accounts']) == 1:
            d['accounts'] = d['accounts'][0]
        output.append(d)

print(output)

Output:

[{'name': 'core', 'accounts': ['foo-bar', 'zed-bar'], 'options': {'foo': 'bar'}}, 
{'name': 'core', 'accounts': 'tup-bar', 'options': {'foo': 'zed'}}]
user_31445000
Related