Implimentation of WARP loss in Keras

Question

I am trying to implement warp loss (type of pairwise ranking function) with Keras API. I am kinda stuck how this can be succeeded.

The definition of warp loss is taken from lightFM doc.:

For a given (user, positive item pair), sample a negative item at random from all the remaining items. Compute predictions for both items; if the negative item’s prediction exceeds that of the positive item plus a margin, perform a gradient update to rank the positive item higher and the negative item lower. If there is no rank violation, continue sampling negative items until a violation is found.

Warp function is used for example in semantic embeddings of #hashtags, a paper published from facebook AI research. In this paper they try to predict the most representable hashtags for short texts. Where 'user' is considered the short text, 'positive item' is the hashtag of the short text, and negative items are some random hashtags uniformly sampled from the 'hashtag lookup'.

I am following the implimentation of another triplet loss to create the warp one: github

My understanding is that for each data point I will have 3 inputs. Example with embeddings('semi' pseudocode):

sequence_input = Input(shape=(100, ), dtype='int32') # 100 features per data point
positive_example = Input(shape=(1, ), dtype='int32', name="positive") # the one positive example
negative_examples = Input(shape=(1000,), dtype='int32', name="random_negative_examples") # 1000 random negative examples.

#map data points to already created embeddings
embedded_seq_input = embedded_layer(sequence_input)
embedded_positive = embedded_layer(positive_example)
embedded_negatives = embedded_layer(negative_examples)

conv1 = Convolution1D(...)(embeddded_seq_input)
               .
               .
               .
z = Dense(vector_size_of_embedding,activation="linear")(convN)

loss = merge([z, embedded_positive, embedded_negatives],mode=warp_loss)
                         .
                         .
                         .

where warp_loss is(where I am assuming of getting 1000 random negative instead of taking all of them and the scores comes of the cosinus similatiry):

def warp_loss(X):
    # pseudocode
    z, positive, negatives = X
    positive_score = cosinus_similatiry(z, positive)
    counts = 1
    loss = 0
    for negative in negatives:
        score = cosinus_similatiry(z, negative)
        if score > positive_score:
           loss = ((number_of_labels - 1) / counts) * (score + 1 - positive_score
        else:
           counts += 1
    return loss

How to compute the warp is described nicely: post

I am not sure if it is the correct way of doing it but i couldn't find a way to implement the warp_loss pseudo function. I can compute cosinus using merge([x,u],mode='cos') but this assumes same dimensions. So I am not sure how to use merge mode cos for the multiple negative examples so I am trying to create my own warp_loss.

Any insights, implemented similar examples, comments are useful.

user_304926 · Answer

First of all, I would argue that it is not possible to implement WARP in the batch training paradigm. Therefore you can't implement WARP in Keras. This is because WARP is intrinsically sequential, so it can't handle data broken into batches, a la Keras. I suppose if you did fully stochastic batches, you could pull it off.

Typically for WARP you include a margin of 1, but as in the paper you can consider it a hyperparam:

if neg_score > pos_score-1: #margin of 1
  loss = log(num_items / counts) #loss weighted by sample count
  loss = max(1, loss) #this looks like same thing you were doing in diff way

This is superior to it's predecessor BPR, in that optimizes for top k precision instead of average precision.

Implimentation of WARP loss in Keras

1 Answers