Definition of spearmanr

user_5085783 0
Viewed 49

To test my understanding on spearmanr and pearsonr, I compared two ways to compute spearmanr, which should give the same result. Surprisingly, the results are different.

import torch
from scipy.stats import pearsonr, spearmanr

x = torch.normal(1, 1, (10,))
y = torch.normal(1, 1, (10,))
_, x_rank = x.sort()
_, y_rank = y.sort()

print(
    spearmanr(x, y),
    pearsonr(x_rank, y_rank)
)

To reproduce the results, please use the following x and y. This should give a pearsonr of rank of 0.263 and spearmanr of 0.139.

x = torch.tensor([ 1.7443, -0.7889,  0.4698,  1.2080,  0.8847, -0.4490,  1.2561,  1.5188,
        -1.0031,  1.4753])
y = torch.tensor([ 1.2675,  1.8317,  1.6912, -0.2964,  2.0014,  1.1092,  2.7958,  2.6034,
        -0.0528, -2.2956])

Why are they different? Isn't spearmanr defined as the pearsonr over the rank of x and that of y? Is there something I missed?

statistics python
1 Answers

Firstly, just for clarification, Pearson and Spearman correlations are not the same, however they can be equal, for eg. in case of perfectly linear relationship. Former is a measure for linear relationship and the later is a measure for monotonic relationship.

Secondly, tensor sort method does not provide you rank but indices of your raw data before sorting. what you need is the indices of the raw data after sorting but in the order of your raw data i.e. rank.

you can do something like:

import torch
from scipy.stats import pearsonr, spearmanr

x = torch.tensor([ 1.7443, -0.7889,  0.4698,  1.2080,  0.8847, -0.4490,  1.2561,  1.5188,
        -1.0031,  1.4753])
y = torch.tensor([ 1.2675,  1.8317,  1.6912, -0.2964,  2.0014,  1.1092,  2.7958,  2.6034,
        -0.0528, -2.2956])

x_sorted, x_raw_indx = x.sort()
y_sorted, x_raw_indx = y.sort()

x_ranked = [int((x_sorted == value).nonzero(as_tuple=True)[0]) for value in x]
y_ranked = [int((y_sorted == value).nonzero(as_tuple=True)[0]) for value in y]

spearmanr(x, y) , pearsonr(x_ranked, y_ranked)

#(SpearmanrResult(correlation=0.13939393939393938, pvalue=0.7009318849100584),
# (0.1393939393939394, 0.7009318849100588))

As mentioned in comments (@simon), a much cleaner way to get rank is:

x_ranked = x.argsort().argsort()
y_ranked = y.argsort().argsort()
user_82239540
Related