If you don't want to deal with a complex statistical model like the Elo rating system suggested by @VincentRamdhanie (which will yield optimal results), you can always model this as a simple optimization problem.
You have datapoints of the type a>b. If you assign values to a and b, a>b is simply a condition that evaluates to true or false. Then, one possible "solution" to this problem is to maximize the number of conditions (datapoints) that evaluate to true.
Since you already know your metric, all that is left is to choose a search algorithm. Hill climbing is a very simple one. It works like this:
- assign to each rating a random value - this is your current solution
- change the value of one of the ratings
- if by performing the change, our metric (number of valid conditions) is improved, the change is incorporated in our current solution
- repeat until you're happy (typically, until you can't improve the solution for some time)
It's important to mention that this method will yield meaningful order between the images, but the ratings themselves will not have any meaning. There's no difference between a rating of 3 or 4 if there are no images with ratings between 3 or 4. After the search algorithm runs, you may want to simply take the order information, and distribute the images evenly in the rating space (if you have 3 images, they'll have final ratings of 1, 3 and 5, for example).
Here's some code to illustrate this:
import random
N_ELEMS= 5 #number of images
N_RATINGS= 5 #number of (faked) user-given ratings
MIN_SCORE, MAX_SCORE= 0.0, 5.0
N_ITERATIONS= 1000 #for search stopping condition
def random_score():
#generate random score between MIN_SCORE and MAX_SCORE
return MIN_SCORE+random.random()*(MAX_SCORE-MIN_SCORE)
elements=range(N_ELEMS) #this would be strings or objects in the real world
ratings= [] #tuples of (elem_a, elem_b), representing rating a<b
for i in range(N_RATINGS):
#generate fake ratings
while True:
elem_a, elem_b=(random.choice(elements),random.choice(elements))
if elem_a!=elem_b:
break
ratings.append((elem_a, elem_b))
scores= [random_score() for i in range(N_ELEMS)] #assign random scores
def evaluate_condition( rating ):
#is a user-provided rating true, given the current scores
return scores[rating[0]]<scores[rating[1]]
def metric():
#number of true conditions
return sum( map(evaluate_condition, ratings))
no_improvement_iterations=0 #number of successive iterations where there has been no improvement
current_score= metric()
while no_improvement_iterations<N_ITERATIONS:
change_element= random.randint(0,N_ELEMS-1)
new_value= random_score()
old_value= scores[change_element]
scores[change_element]= new_value
new_score= metric()
if new_score<=current_score:
scores[change_element]= old_value
no_improvement_iterations+=1
else:
no_improvement_iterations=0
current_score= new_score
def distribute_scores(scores):
'''distribute scores evently in the interval (MIN_SCORE, MAX_SCORE)'''
sorted_scores= sorted(scores)
order= [sorted_scores.index(x) for x in scores] #inefficient but easy to understand
step= (MAX_SCORE-MIN_SCORE)/(len(order))
return [x*step for x in order]
print "ratings:", ", ".join(["{0}<{1}".format(a,b) for a,b in ratings])
print "scores:", scores
print "distributed scores:", distribute_scores(scores)
And output:
ratings: 1<2, 3<4, 0<3, 4<3, 4<0
scores: [2.3647080073611955, 0.7188260611863462, 4.295792794993049, 4.286501742802684, 0.3471914376983337]
distributed scores: [2.0, 1.0, 4.0, 3.0, 0.0]
This is not strictly hill climbing, since we generate new_value randomly. Hill climbing would choose new_value to maximize the score, but we can't calculate that directly, so we use random optimization
Also, obviously, this is not the best way to solve the search problem - a genetic algorithm that operated on the order would probably be faster, for example. I aimed for simplicity, not efficiency.
