How to filter map content by path

Viewed 181

I want to select paths of a deeply nested map to keep.

For example:

{:a 1
 :b {:c [{:d 1 :e 1} 
         {:d 2 :e 2}]
     :f 1}
 :g {:h {:i 4 :j [1 2 3]}}}

I want to select by paths, like so:

(select-paths m [[:a] 
                 [:b :c :e]
                 [:b :f]
                 [:g :h :i]])

This would return

{:a 1
 :b {:c [{:e 1}
         {:e 2}]
     :f 1}
 :g {:h {:i 4}}}

Essentially the same as Elasticsearch's fields parameter. The format of the paths argument can be something else, this is just the first idea.

I tried two different solutions

  1. Go through the entire map and checking if the full path of the current element is in the given paths. I can't figure out how to handle lists of maps so that they are kept as lists of maps.
  2. Creating select-keys statements from the given paths but again I run into problems with lists of maps - and especially trying to resolve paths of varying depths that have some common depth.

I looked at spectre but I didn't see anything that would do this. Any map or postwalk based solution I come up with turns into something incredibly convoluted at some point. I must be thinking about this the wrong way.

If there's a way to do this with raw json, that would be fine as well. Or even a Java solution.

2 Answers

There is no simple way to accomplish your goal. The automatic processing implied for the sequence under [:b :c] is also problematic.

You can get partway there using the Tupelo Forest library. See the Lightning Talk video from Clojure/Conj 2017.

I did some additional work in data destructuring that you may find useful building the tupelo.core/destruct macro (see examples here). You could follow a similar outline to build a recursive solution to your specific problem.

A related project is Meander. I have worked on my own version which is like a generalized version of tupelo.core/destruct. Given data like this

(def skynet-widgets [{:basic-info   {:producer-code "Cyberdyne"}
                      :widgets      [{:widget-code      "Model-101"
                                      :widget-type-code "t800"}
                                     {:widget-code      "Model-102"
                                      :widget-type-code "t800"}
                                     {:widget-code      "Model-201"
                                      :widget-type-code "t1000"}]
                      :widget-types [{:widget-type-code "t800"
                                      :description      "Resistance Infiltrator"}
                                     {:widget-type-code "t1000"
                                      :description      "Mimetic polyalloy"}]}
                     {:basic-info   {:producer-code "ACME"}
                      :widgets      [{:widget-code      "Dynamite"
                                      :widget-type-code "c40"}]
                      :widget-types [{:widget-type-code "c40"
                                      :description      "Boom!"}]}])

You can search and extract data using a template like this:


    (let [root-eid (td/add-entity-edn skynet-widgets)
          results  (td/match
                     [{:basic-info   {:producer-code ?}
                       :widgets      [{:widget-code      ?
                                       :widget-type-code wtc}]
                       :widget-types [{:widget-type-code wtc
                                       :description      ?}]}])]
      (is= results
        [{:description "Resistance Infiltrator" :widget-code "Model-101" :producer-code "Cyberdyne" :wtc "t800"}
         {:description "Resistance Infiltrator" :widget-code "Model-102" :producer-code "Cyberdyne" :wtc "t800"}
         {:description "Mimetic polyalloy" :widget-code "Model-201" :producer-code "Cyberdyne" :wtc "t1000"}
         {:description "Boom!" :widget-code "Dynamite" :producer-code "ACME" :wtc "c40"}])))

This code is working (see here) but it needs more polish. You could use it as a guide to building a generalized select-paths function.


Can you add any details on how this problem arose or the specific context? That may point to ideas for an alternate solution.

One way of solving this problem would be to generate a set of all subpaths that you accept and then write a recursive function that traverses the data structure and keeps track of the path to the current node. The code that accomplishes that does not need to be very long:

(defn select-paths-from-set [current-path path-set data]
  (cond
    (map? data) (into {}
                      (remove nil?)
                      (for [[k v] data]
                        (let [p (conj current-path k)]
                          (if (contains? path-set p)
                            [k (select-paths-from-set p path-set v)]))))
    (sequential? data) (mapv (partial select-paths-from-set current-path path-set) data)
    :default data))

(defn select-paths [data paths]
  (select-paths-from-set []
                         (into #{}
                               (mapcat #(take-while seq (iterate butlast %)))
                               paths)
                         data))

(select-paths {:a 1
               :b {:c [{:d 1 :e 1} 
                       {:d 2 :e 2}]
                   :f 1}
               :g {:h {:i 4 :j [1 2 3]}}}
              [[:a] 
               [:b :c :e]
               [:b :f]
               [:g :h :i]])
;; => {:a 1, :b {:c [{:e 1} {:e 2}], :f 1}, :g {:h {:i 4}}}
Related