Merged two json data, but don't copy

user_11561146 0
Viewed 47

I want to merged two json data, which I did. Please see the python code and json data below.

json_data

 [
  {
    "fields": {
      "Full Address": "data1",
      "hz": "text1",
      "ot": "doc1"
   }
  },
  {
  "fields": {
    "Full Address": "data2",
    "hz": "text2",
    "ot": "doc2"
  }
 }
 ]

json_data1

 [
  {
    "fields": {
      "Full Address": "data2",
      "hz": "text2",
      "ot": "doc2"
   }
  },
  {
  "fields": {
    "Full Address": "data3",
    "hz": "text3",
    "ot": "doc3"
  }
 }
 ]

Result_merged_json

 [
  {
    "fields": {
      "Full Address": "data1",
      "hz": "text1",
      "ot": "doc1"
   }
  },
  {
  "fields": {
    "Full Address": "data2",
    "hz": "text2",
    "ot": "doc2"
  }
 },
  {
    "fields": {
      "Full Address": "data2",
      "hz": "text2",
      "ot": "doc2"
   }
  },
  {
  "fields": {
    "Full Address": "data3",
    "hz": "text3",
    "ot": "doc3"
  }
 }
 ]

Python_code

 import json
 import io

 with open("json1.json") as f1:
  data = json.load(f1)

 with open("json2.json") as f2:
  data1 = json.load(f2)

 data.extend(data1)

 with io.open('Result_merged_json.json', 'w', encoding='utf8') as f3: 
     data = json.dumps(data, indent=2, ensure_ascii=False)
   f3.write(data)

Here in the result: "Full Address": "data2" along with the other values comes two time. But I want one time in my Result_merged_json. I tried to do some changing in code, but I could not get result as per my choice. So how could I preventing from copying the data?

  {
  "fields": {
    "Full Address": "data2",
    "hz": "text2",
    "ot": "doc2"
   }
  }
merge python json
2 Answers

You can merge by an unique key (for example 'Full Address'):

>>> from operator import getitem
>>> from functools import reduce
>>> nested_getitem = lambda data, keys: reduce(getitem, keys, data)
>>> unique, merged_json = set(), list()

>>> for dict_item in json_data1 + json_data2:
        key = nested_getitem(dict_item, ['fields', 'Full Address'])
        if key not in unique:
            unique.add(key)
            merged_json.append(dict_item)

>>> merged_json
[{'fields': {'Full Address': 'data1', 'hz': 'text1', 'ot': 'doc1'}},
 {'fields': {'Full Address': 'data2', 'hz': 'text2', 'ot': 'doc2'}},
 {'fields': {'Full Address': 'data3', 'hz': 'text3', 'ot': 'doc3'}}]

If you want to check by all the values of fields:

>>> unique, merged_json = set(), list()

>>> for dict_item in json_data1 + json_data2:
        keys = tuple(dict_item['fields'].values())
        if keys not in unique:
            unique.add(keys)
            merged_json.append(dict_item)

>>> merged_json
[{'fields': {'Full Address': 'data1', 'hz': 'text1', 'ot': 'doc1'}},
 {'fields': {'Full Address': 'data2', 'hz': 'text2', 'ot': 'doc2'}},
 {'fields': {'Full Address': 'data3', 'hz': 'text3', 'ot': 'doc3'}}]

# In this case, unique looks like this:
>>> unique
{('data1', 'text1', 'doc1'),
 ('data2', 'text2', 'doc2'),
 ('data3', 'text3', 'doc3')}
user_54317910

Assumed that you need to remove the duplicated item which is TOTALLY SAME

...
data1.extend(data2)
# add the following statement
data1 = list(map(lambda dct: eval(dct), set(map(lambda dct: str(dct), data1))))
...

The code is ugly but useful :)

user_153799530
Related