How to remove duplicate objects from a json stream with jq without reordering?

user_1687259 0
Viewed 112

Here's your input data: "3""5""2""2".
Figure out a jq query to return "3""5""2".
My usecase: polling JSON objects of a third party service state, which changes less often than I'm polling.

I tried jq -n '[inputs] | unique[]', but this reorders results.

jq unique stream json
1 Answers

This is covered in the jq Cookbook. Here is an extract, omitting two defs that are unnecessary for defining uniques/1:

Using bag to implement a sort-free version of unique

jq's unique built-in involves a sort, which in practice is usually fast enough, but may not be desirable for very large arrays or especially if processing a very long stream of entities, or if the order of first-occurrence is important. One solution is to use "bags", that is, multisets in the sense of sets-with-multiplicities. Here is a stream-oriented implementation that preserves generality and takes advantage of jq's implementation of lookups in JSON objects:


# bag(stream) uses a two-level dictionary: .[type][tostring]
# So given a bag, $b, to recover a count for an entity, $e, use
# $e | $b[type][tostring]
def bag(stream):
  reduce stream as $x ({}; .[$x|type][$x|tostring] += 1 );

It is now a simple matter to define uniques(stream), the "s" being appropriate here because the filter produces a stream:


# Produce a stream of the distinct elements in the given stream
def uniques(stream):
  bag(stream)
  | to_entries[]
  | .key as $type
  | .value
  | to_entries[]
  | if $type == "string" then .key else .key|fromjson end ;
user_9973580
Related