Remove duplicate JSON object in file using C#

user_4817250 0
Viewed 4088

I have few thousand json objects in a file. I would want to clean it by removing duplicate entry by id.

JSON File Structure:

{
"quote": [{
"id":1,
"title": "Mahatma Gandhi",
"quote":"A man is but a product of his thoughts. What he thinks he becomes.",
"videourl": "https://www.youtube.com/watch?v=2GgK_Nq9NLw",
"websiteurl":"http://www.mkgandhi.org/",
"otherurl":"https://en.wikipedia.org/wiki/Mahatma_Gandhi",
"updatedon":"19-Aug-2017",
"username":"XXX@gmail.com",
"details":"Mohandas Karamchand Gandhi (2 October 1869 – 30 January 1948), commonly known as Mahatma Gandhi (Sanskrit: महात्मा mahātmā 'Great Soul'). In India he is generally regarded as Bapu (Gujarati: બાપુ bāpu 'father'), Jathi Pitha and Raashtra Pita; he was an advocate and pioneer of nonviolent social protest and direct action in the form he called Satyagraha. He led the struggle for India's independence from British colonial rule."
},
{
"id":2,
"title": "Ernest Hemingway",
"quote":"The world breaks everyone, and afterward, some are strong at the broken places.",
"videourl": "https://www.youtube.com/watch?v=35vrg7a64nY",
"websiteurl":"https://www.biography.com/people/ernest-hemingway-9334498",
"otherurl":"https://en.wikipedia.org/wiki/Ernest_Hemingway",
"updatedon":"20-Aug-2017",
"username":"XXX@gmail.com",
"details":"Ernest Miller Hemingway was an American novelist, short story writer, and journalist. His economical and understated style had a strong influence on 20th-century fiction, while his life of adventure and his public image influenced later generations."
},
...
]

I created a Quote entity and deserialize object to find duplicates using Linq and write the stream back, in C#. It take considerable amount of time to complete.

Please suggest any other effective method you know which works well.

Edit: Enclosed the code I tried,

string json = "";

using (StreamReader file = File.OpenText(@"D:\motivation-pyramid\data\demo\quote.json"))
{
    json = file.ReadToEnd();
}

var managedObjectCollection = JsonConvert.DeserializeObject<ManagedObjectCollection>(json);

var distinctManagedObjects = managedObjectCollection.quote.GroupBy(a => a.quote).Select(b => b.First());

json = JsonConvert.SerializeObject(distinctManagedObjects.ToArray());

File.WriteAllText(@"D:\motivation-pyramid\data\demo\quote.json", json);
bigdata json c#
2 Answers

Using Netwonsoft Json.NET library, this can be achieved pretty easily:

JObject jObj = JObject.Parse(json_string);

List<JToken> uni = jObj["quote"].GroupBy(x => x["id"]).Select(x => x.First()).ToList();

for (Int32 i = jObj["quote"].Count() - 1; i >= 0; --i)
{
    JToken token = jObj["quote"][i];

    if (!uni.Contains(token))
        token.Remove();
}

If you don't want that library, once your Json data is serialized, into a list you can use the following LINQ expression:

list.DistinctBy(x => x.ID).ToList();

Alternatively, with MoreLINQ:

list.DistinctBy(x => x.ID).ToList();
user_7960850

"Normally" one would use property names of the class, however refactoring a class or in our case obfuscating will/can cause production issues after deploying updates.

I, therefore, had a similar issue where had to manually set the property names of classes, when one has lots of properties one tends to get copy past errors and names like [JsonProperty("A1")]..[JsonProperty("Z9")] tend to get errors in it, we asked the folks at PostSharp if they have I can generate a compiler error when I have duplicate names using their post processor.

Here is their answer:

public class TestValidator : TypeLevelAspect
{
    public override bool CompileTimeValidate(Type type)
    {
        HashSet<string> propertyNames = new HashSet<string>();

        foreach (PropertyInfo property in type.GetProperties(BindingFlags.Public | BindingFlags.Instance))
        {
            JsonPropertyAttribute jsonProperty = (JsonPropertyAttribute)property.GetCustomAttribute(typeof(JsonPropertyAttribute));

            if (jsonProperty == null)
                continue;

            if (!propertyNames.Add(jsonProperty.PropertyName))
            {
                Message.Write(property, SeverityType.Error, "A01", "Duplicate JSON property {0} specified on {1}.",
                    jsonProperty.PropertyName, property);
            }
        }

        return false;
    }
}

Now I get compiler errors when i have duplicates simply by adding the aspect to the class, problem solved :-)

Have a look at PostSharp, they have lost of nice patterns, and you can make your own as well.

user_80003820
Related