How to use | add and preserve duplicate keys with different values, adding them to an array instead

Question

I am attempting to take two similar JSON objects and combine them because they represent two versions of the same thing. So far I have a fairly simple solution, by simply using "add." However, that eliminates duplicate keys that have different values and apparently the values chosen are chosen randomly. My desired outcome is that if the values of a key exist in both places but are different, they should be put into an array and that array would then be the value of that key. Any keys that exist in one but not the other should also be added. Lastly, for any key that exists in both places, where the value is the same, there should only be one key/value for that item. For example:

Object A:

{ 
    "primaryKey":"thisValueIsTheSameInBoth",
    "aOnlyField":"thisIsOnlyInA",
    "sharedField":"thisIsInBoth-AAA",
    "sharedField2":"thisIsInBothAgain-AAA"
}

Object B:

{
    "primaryKey":"thisValueIsTheSameInBoth",
    "bOnlyField":"thisIsOnlyInB",
    "sharedField":"thisIsInBothButDifferentInB",
    "sharedField2":"thisIsInBothAgainButStillDifferentInB"
}

Currently I am doing this:

 | jq -n '[inputs] | add'

Unfortunately that results in this:

{
    "primaryKey":"thisValueIsTheSameInBoth",
    "aOnlyField":"thisIsOnlyInA",
    "bOnlyField":"thisIsOnlyInB",
    "sharedField":"thisIsInBoth-AAA",
    "sharedField2":"thisIsInBothAgain-AAA"
}

As you can see, it does a good job of combining fields that don't exist in both, but arbitrarily decides to pick the value of A for all the shared fields that differ. I'd like to put those differering values in an array instead of a single string. Is there a clean, easy way to do that?

Desired output:

{
    "primaryKey":"thisValueIsTheSameInBoth",
    "aOnlyField":"thisIsOnlyInA",
    "bOnlyField":"thisIsOnlyInB",
    "sharedField" : [ "thisIsInBoth-AAA", "thisIsInBothButDifferentInB" ],
    "sharedField2" : [ "thisIsInBothAgain-AAA", "thisIsInBothAgainButStillDifferentInB" ]
}

user_16332641 · Answer

The following code does what you are looking for.

The idea is to collect the values for all keys in arrays in the first step (also for keys that occur only once) and at the end remove duplicates from all arrays and unpack the arrays with only one element.

This may be an unperformant solution for a large number of objects.

#!/bin/bash     

object_a='
{
    "primaryKey":"thisValueIsTheSameInAll",
    "aOnlyField":"thisIsOnlyInA",
    "sharedField":"thisIsInAllWithSameValue",
    "sharedField2":"thisIsInAllButDifferentValue-A",
    "thing":["hello","hi"]
}'

object_b='
{
    "primaryKey":"thisValueIsTheSameInAll",
    "bOnlyField":"thisIsOnlyInB",
    "sharedField":"thisIsInAllWithSameValue",
    "sharedField2":"thisIsInAllButDifferentValue-B",
    "thing":["hola","bonjour"]
}'

object_c='
{
    "primaryKey":"thisValueIsTheSameInAll",
    "sharedField":"thisIsInAllWithSameValue",
    "sharedField2":"thisIsInAllButDifferentValue-C"
}'

echo "$object_a $object_b $object_c" |
jq -n '
  reduce inputs as $input
         ({}; reduce ($input | keys_unsorted)[] as $key
                     (.; .[$key] += [$input[$key]])
         )
  | with_entries(.value |= (flatten
                            | unique
                            | if length == 1 then .[0] else . end)         
                )'

result:

{
  "primaryKey": "thisValueIsTheSameInAll",
  "aOnlyField": "thisIsOnlyInA",
  "sharedField": "thisIsInAllWithSameValue",
  "sharedField2": [
    "thisIsInAllButDifferentValue-A",
    "thisIsInAllButDifferentValue-B",
    "thisIsInAllButDifferentValue-C"
  ],
  "thing": [
    "bonjour",
    "hello",
    "hi",
    "hola"
  ],
  "bOnlyField": "thisIsOnlyInB"
}

How to use | add and preserve duplicate keys with different values, adding them to an array instead

1 Answers