Is there an efficient way of clustering labelled similar datapoints?

user_15942112 0
Viewed 65

I have a data frame that looks like this:

| Image | Similar Images |
| ------| -------------- |
| 1     | [1, 2, 6]      |
| 2     | [2, 1, 6]      |
| 3     | [3, 4]         |
| 4     | [4, 3]         |
| 5     | [5]            |
| 6     | [6, 1, 2]      |

And I want to make clusters of similar images and label them. What I aim to do would look something like this:

| Image | Similar Images | Label |
| ------| -------------- |-------|
| 1     | [1, 2, 6]      | 1     |
| 2     | [2, 1, 6]      | 1     |
| 3     | [3, 4]         | 2     |
| 4     | [4, 3]         | 2     |
| 5     | [5]            | 3     |
| 6     | [6, 1, 2]      | 1     |

Is there an efficient way to do this? I have limited computing resources and around 178000 images, which is why I'm wondering if there are any efficient existing methods or packages that could perform (part of) this task.

pandas cluster-analysis search python
2 Answers

Try frozenset then factorize

df['new'] = df['Similar Images'].map(frozenset).factorize()[0]+1
user_79645270

If you don't insist to use incremental integers for the labels，you can just sort the Similar Images list and calculate its hash, and use that hash for the labels

from hashlib import sha1

images = [
  [1, 2, 6],
  [2, 1, 6],
  [3, 4],
  [4, 3], 
  [5],
  [6, 1, 2]
]

labels = [sha1(str(sorted(item)).encode()).hexdigest() for item in images]

print(labels)

#['c43178906089cf513e06bd8e70af5323c44aeac9',
# 'c43178906089cf513e06bd8e70af5323c44aeac9',
# '848eeae1a32f2e1c3da6f07361b19f7b35913dfd',
# '848eeae1a32f2e1c3da6f07361b19f7b35913dfd',
# '10ae24979c5028fa873651bca338152dc0484245',
# 'c43178906089cf513e06bd8e70af5323c44aeac9']
user_130438710
Related