Find the optimal way to calculate the highest possible Pearson correlation for millions of combinations in R

user_13932170 0
Viewed 114

I am working on a large dataset. However I will start with a small example to illustrate what I am trying to achieve.

I have the following vectors:

season <- c("2019")
round <- c("1")
team <- c("Team A", "Team B", "Team C")
margin <- c(33, 56, 63)
score_A <- c(0.330, 0.256, 0.118)
score_B <- c(0.584, 0.176, 0.342)
score_C <- c(0.118, 0.193, 0.286)

And I create a data frame like so:

df1 <- data.frame(season, round, team, score_A, score_B, score_C, margin)

I then apply weightings to each of the scores like:

df1$score_A <- df1$score_A * 0.25
df1$score_B <- df1$score_B * 0.5
df1$score_C <- df1$score_C * 0.75

I then sum all the scores and create a team total score:

df1$score_total <- df1$score_A + df1$score_B + df1$score_C

library(dplyr)
df1 <- df1%>%group_by(season, round, team)%>%dplyr::mutate(score_Team_Total=sum(score_total))

I know that I can then calculate the Pearson like so:

> cor(df1$margin, df1$score_Team_Total, method = "pearson")
[1] -0.5505451

Although this does not give me a line by line return, I am not quite sure yet how to calculate that.

However this is where it starts to get tricky.

I have a number of weightings that I would like to apply to each of the scores like so:

weightings <- c(0.25,0.5,0.75,1,1.25,1.5,2,2.5,3)

And I have a number of scores for each weighting (score_A right through to score_R).

The first combination would be:

df1$score_A <- df1$score_A * 0.25
df1$score_B <- df1$score_B * 0.25
df1$score_C <- df1$score_C * 0.25

The second combination would be:

df1$score_A <- df1$score_A * 0.25
df1$score_B <- df1$score_B * 0.25
df1$score_C <- df1$score_C * 0.5

The third combination would be:

df1$score_A <- df1$score_A * 0.25
df1$score_B <- df1$score_B * 0.5
df1$score_C <- df1$score_C * 0.5

And so on.

But how can I get the Pearson correlation for each combination and return the highest possible Pearson?

I know there will be millions of comibations, as I ran this:

> length(permutations(7, 9, repeats.allowed = TRUE))
[1] 363182463

But I have 9 different variables in my weightings (0.25,0.5,0.75,1,1.25,1.5,2,2.5,3) and 18 different scores (score_A through to score_R).

So when I tried:

> length(permutations(9, 18, repeats.allowed = TRUE))

I received this error:

Error: cannot allocate vector of size 73.7 Gb

So I know the number is going to be very large.

I need to apply each combination of weightings to the scores, then create the totals and calculate the Pearson.

A dataframe or list with the results would be too large I assume, so is there a way to return the optimal comination? The output would look something like:

            score_A   score_B   score_C   pearson
weighting     0.25      0.50      0.25    0.63

I am still new to R and learning so I am not quite sure where to go from here.

r
1 Answers

You should realize that you are trying to explore 9^18 permutations, that is:

options(scipen = 999)
9^18
# [1] 150094635296999136

What about exploring a subset of them? The following code generates 18^7 combinations of your weightings:

set.seed(1)
n_scores <- 18
p <- 7
aux <- matrix(sample(weightings, n_scores^p, replace = TRUE), ncol = n_scores)
# First combination
aux[1,]
 [1] 3.00 2.00 0.50 1.00 1.25 2.00 1.50 2.50 0.25 0.75 3.00 2.50 1.25 3.00
[15] 0.75 2.50 0.50 0.50

Then, you could repeat this smaller exploration several times and look at the similarity of the several optimal combinations to gain some insights.

Edit upon the comment by @Michael:

First, I modify your toy example to have an extra row:

season <- c("2019")
round <- c("1")
team <- c("Team A", "Team B", "Team C", "Team D")
margin <- c(33, 56, 63, 50)
score_A <- c(0.330, 0.256, 0.118, 0.2)
score_B <- c(0.584, 0.176, 0.342, 0.15)
score_C <- c(0.118, 0.193, 0.286, 0.2)
df1 <- data.frame(season, round, team, score_A, score_B, score_C, margin)

Then, I generate 9 sets of weights:

weightings <- c(0.25,0.5,0.75,1,1.25,1.5,2,2.5,3)
set.seed(1)
n_scores <- 3
p <- 3
aux1 <- matrix(sample(weightings, n_scores^p, replace = TRUE), ncol = n_scores)
colnames(aux1) <- c("score_A", "score_B", "score_C")

Finally, I perform the main operation

aux2 <- cbind(df1$score_A, df1$score_B, df1$score_C)
df2 <- data.frame(aux1, 
                  pearson = c(cor(df1$margin, apply(aux1, 1, function(x) rowSums(t(x*t(aux2)))))))
df2
#   score_A score_B score_C    pearson
# 1    3.00    1.25    1.25 -0.8473964
# 2    1.00    1.25    1.25 -0.6385250
# 3    2.00    1.50    0.50 -0.8222945
# 4    0.25    2.00    3.00 -0.2510155
# 5    0.50    3.00    0.25 -0.6804298
# 6    2.00    1.25    1.00 -0.8025296
# 7    0.50    1.25    0.75 -0.6260844
# 8    0.75    3.00    1.50 -0.6088807
# 9    0.25    3.00    1.50 -0.5591034

Edit upon the 2nd comment by @Michael:

After creating aux2 as above, generate aux3 as below. In aux3 you will have as many columns called w_x as the number of sets of weights that you are exploring, but also the original columns from df1 that you should need for your next computations. Each w_x is a weighted sum of the scores:

aux3 <- apply(aux1, 1, function(x) rowSums(t(x*t(aux2))))
colnames(aux3) <- paste0("w_", 1:ncol(aux3))
df1 %>%
  select(season, round, team, margin) %>%
  cbind(aux3) -> aux3
aux3
#   season round   team margin     w_1     w_2    w_3    w_4     w_5    w_6
# 1   2019     1 Team A     33 1.86750 1.20750 1.5950 1.6045 1.94650 1.5080
# 2   2019     1 Team B     56 1.22925 0.71725 0.8725 0.9950 0.70425 0.9250
# 3   2019     1 Team C     63 1.13900 0.90300 0.8920 1.5715 1.15650 0.9495
# 4   2019     1 Team D     50 1.03750 0.63750 0.7250 0.9500 0.60000 0.7875
#       w_7    w_8    w_9
# 1 0.98350 2.1765 2.0115
# 2 0.49275 1.0095 0.8815
# 3 0.70100 1.5435 1.4845
# 4 0.43750 0.9000 0.8000
user_127233230
Related