Whats the 'Ruby way' to identify duplicate values in a hash?

Suppose

original_hash = {:a=>1, :b=>2, :c=>2, :d=>2, :e=>3, :f=>4, :g=>4}

If you were only interested in returning a hash uniques_hash that contained unique values, you could write the following.

uniques_hash = original_hash.invert.invert
  #=> {:a=>1, :d=>2, :e=>3, :g=>4}

the intermediate step being

original_hash.invert
  #=> {1=>:a, 2=>:d, 3=>:e, 4=>:g}

See Hash#invert. Note that uniques_hash, as defined, is not itself unique. It could be any of the following.

{:a=>1, :b=>2, :e=>3, :f=>4}
{:a=>1, :b=>2, :e=>3, :g=>4}
{:a=>1, :c=>2, :e=>3, :f=>4}
{:a=>1, :c=>2, :e=>3, :g=>4}
{:a=>1, :d=>2, :e=>3, :f=>4}
{:a=>1, :d=>2, :e=>3, :g=>4}

Another way of doing this is to use Enumerable#uniq and Array#to_h.

unique_hash = original_hash.uniq(&:last).to_h
  #=> {:a=>1, :b=>2, :e=>3, :f=>4} 

the intermediate calculation being

original_hash.uniq(&:last)
  #=> [[:a, 1], [:b, 2], [:e, 3], [:f, 4]]

which is shorthand for

original_hash.uniq { |_k,v| v }

Presumably, each key of duplicates_hash is a value in original_hash and the value of that key is an array of those keys k in original_hash for which original_hash[k] == v.

One way to compute duplicates_hash is as follows.

duplicates_hash = original_hash.each_with_object({}) do |(k,v),h|
  h[v] = (h[v] || []) << k
end
  #=> {1=>[:a], 2=>[:b, :c, :d], 3=>[:e], 4=>[:f, :g]}

This can also be written

duplicates_hash = original_hash.
  each_with_object(Hash.new { |h,k| h[k] = [] }) { |(k,v),h| h[v] << k }
  #=> {1=>[:a], 2=>[:b, :c, :d], 3=>[:e], 4=>[:f, :g]}

See Hash::new. Both forms are equivalent to

duplicates_hash = original_hash.each_with_object({}) do |(k,v),h|
  h[v] = [] unless h.key?(v)
  h[v] << k
end

Writing the block variables as |(k,v),h| makes use of array decomposition.

We have an enumerator that will generate values and pass them to its block.

enum = original_hash.each_with_object({})
  #=> #<Enumerator: {:a=>1, :b=>2, :c=>2, :d=>2, :e=>3, :f=>4, :g=>4}:
  #     each_with_object({})> 

Enumerators are instances of the class Enumerator.

The first value of the enumerator is generated and the block variables are assigned values like so:

(k,v),h = enum.next
  #=> [[:a, 1], {}]

Array decomposition is seen to split this array of two elements as follows:

k #=> :a 
v #=> 1 
h #=> {} 

Notice how the parentheses on the left correspond to the inner brackets on the right. The block calculation is then performed using these variables.

h[v] = (h[v] || []) << k
  #=> [:a]

Now,

h #=> {1=>[:a]}

The next value is then generated by the enumerator and the block calculation is performed.

(k,v),h = enum.next
  #=> [[:b, 2], {1=>[:a]}] 
k #=> :b 
v #=> 2 
h #=> {1=>[:a]} 
h[v] = (h[v] || []) << k

so now

h #=> {1=>[:a], 2=>[:b]} 

This continues until

enum.next
  #=> Stop Interation (exception)

causing Ruby to return the value of h.

Note that by computing duplicates_hash first we could compute uniques_hash as follows.

keeper_keys = duplicates_hash.values.map(&:first)
  #=> [:a, :b, :e, :f] 
unique_keys = original_hash.slice(*keeper_keys)
  #=> {:a=>1, :b=>2, :e=>3, :f=>4} 

or

unique_keys = original_hash.slice(*duplicates_hash.values.map(&:first))
  #=> {:a=>1, :b=>2, :e=>3, :f=>4}

See Hash#slice. If one feels guilty by favouring certain keys one could instead write

unique_keys = original_hash.slice(*duplicates_hash.values.map(&:sample))
  #=> {:a=>1, :b=>2, :e=>3, :g=>4}

See Array#sample.

it might or might not be the best way to go about this, but I've used the "group_by" and "select" functions to get me a new hash that finds duplicates:

hash.group_by{|k,v| v}.select{|k,v| v.count > 1}

in this case, the returned hash will look a bit like:

{value: [{key: value}, {key: value}]}

Count the values using group_by and store the results in a hash. Use that hash to partition the original hash like so:

h = {a: 1, b: 2, c: 2, d: 2, e: 3, f: 4, g: 4}

cnt = Hash[h.values.group_by{ |i| i }.map { |k, v| [k, v.count] }]
h_uniq, h_dups = h.partition{ |k, v| cnt[v] == 1 }.map(&:to_h)

puts cnt
# {1=>1, 2=>3, 3=>1, 4=>2}

puts h_uniq.inspect
# {:a=>1, :e=>3}

puts h_dups.inspect
# {:b=>2, :c=>2, :d=>2, :f=>4, :g=>4}

Thanks to all for the help on this! I thought it might help someone if I posted my draft code, and any comments /improvements are very welcome! (I'm in the process of refactoring Listing..) N

#!/usr/bin/env ruby
# shebang to run the script from Terminal

# include shasum
require 'digest'

class Listing
  # class of arrays of file listings 
attr_reader :path
  def initialize(path)
    @path = path
    Dir.chdir(@path)
    @list_of_items = Dir['**/*']
    @list_of_folders = []
    @list_of_files = []
    @list_of_extensions = []
    @list_of_uniques = []
    @list_of_duplicates = []
  end

  def to_s
    "#{@path}"
  end
  
  def analyse_items
    @list_of_items.each do |f|
      if File.directory?(f)
        @list_of_folders << f
      else
        @list_of_files << f
      end
    end
    @list_of_folders.sort!
    @list_of_files.sort!
    @folder_count = @list_of_folders.count
    @files_count = @list_of_files.count
    @items_count = @list_of_items.count
    @count_check = (@items_count -(@folder_count + @files_count))
    # count_check should be zero
  end

  def identify_duplicates
    # Given an array of filepaths, this method divides it into an array of unique files and an array of duplicated files.  
    source = {}
    uniques = {}
        
    @list_of_files.each do |f|
      digest = Digest::SHA512.hexdigest File.read f
      source.store(f, digest)
    end
    
    uniques = source.invert.invert
    @list_of_uniques = uniques.keys
    @list_of_duplicates = source.keys - uniques.keys
  end
  
  def tell_duplicates
    puts "dupes = #{@list_of_duplicates}"
  end
    
end
 


l = Listing.new("/Volumes/Things/Photos/")
l.analyse_items
l.identify_duplicates
l.tell_duplicates