Is there a method in Pytorch to count the number of unique values in a way that can be back propagated?

user_2634895 0
Viewed 11923

Given the following tensor (which is the result of a network [note the grad_fn]):

tensor([121., 241., 125.,   1., 108., 238., 125., 121.,  13., 117., 121., 229.,
        161.,  13.,   0., 202., 161., 121., 121.,   0., 121., 121., 242., 125.],
       grad_fn=<MvBackward>)

Which we will define as:

xx = torch.tensor([121., 241., 125.,   1., 108., 238., 125., 121.,  13., 117., 121., 229.,
        161.,  13.,   0., 202., 161., 121., 121.,   0., 121., 121., 242., 125.]).requires_grad_(True)

I would like to define an operation which counts the number of occurrences of each value in such a way that the operation will output the following tensor:

tensor([2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0,
        0, 7, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0,
        0, 1, 1])

i.e. there are 2 zeros, 1 one, 2 thirteens, etc... the total number of possible values is set upstream, but in this example is 243

So far I have tried the following approaches, which successfully produce the desired tensor, but do not do so in a way that allows computing gradients back through the network:

Attempt 1

tt = []
for i in range(243):
    tt.append((xx == i).unsqueeze(0))
torch.cat(tt,dim=0).sum(dim=1)

Attempt 2

tvtensor = torch.tensor([i for i in range(243)]).unsqueeze(1).repeat(1,xx.shape[0]).float().requires_grad_(True)
(xx==tvtensor).sum(dim=1)

EDIT: Added Attempt

Attempt 3

-- Didn't really expect this to back prop, but figured I would give it a try anyway

ll = torch.zeros((1,243))
for x in xx:
    ll[0,x.long()] += 1

Any help is appreciated

EDIT: As requested the end goal of this is the following:

I am using a technique for calculating structural similarity between two time sequences. One is real and the other is generated. The technique is outlined in this paper (https://link.springer.com/chapter/10.1007/978-3-642-02279-1_33) where a time series is converted to a sequence of code words and the distribution of code words (similar to the way that Bag of Words is used in NLP) is used to represent the time series. Two series are considered similar when the two signal distributions are similar. This is what the counting statistics tensor is for.

What is desired is to be able to construct a loss function which consumes this tensor and measures the distance between the real and generated signal (euclidian norm on the time domain data directly does not work well and this approach claimed better results), so that it can update the generator appropriately.

backpropagation pytorch counting
3 Answers

I would do it with unique method (only to count occurrences):

enter image description here

if you want to count the occurrences, you have to add the parameter return_counts=True

enter image description here

I did it in the version 1.3.1

enter image description here

This is the fast way to count occurrences, however is a non-differentiable operation, therefore, this method is not recommendable (anyway I have described the way to count ocurrences). To perform what you want, I think you should turn the input into a distribution by means of a differentiable function (softmax is the most used) and then, use some way to measure the distance between distributions (output and target) like cross-entropy, KL (kullback-liebler), JS or wasserstein.

user_93907020

You will not be able to do that as unique is simply non-differentiable operation.

Furthermore, only floating point tensors can have gradient as it's defined only for real numbers domain, not for integers.

Still, there might be another, differentiable way to do what you want to achieve, but that's a different question.

user_108864200

The "uniquify" operation is non-differentiable, but there might be ways to remedy this, for instance, by writing a custom operator, or by a clever combination of differentiable operators.

However, you need to ask yourself this question: what do you expect the gradient of such operation to be? Or, on a higher level, what are you trying to achieve with this operation?

user_49092280
Related