Euclidean distant for NON-CONSECUTIVE classes of factors iterated by groups

Question

This question is an extension of this question. Euclidean distant for distinct classes of factors iterated by groups

The same explanations from the previous question apply here as well. I want to calculate the Euclidean distance between consecutive years for each firm based on patent classes according to the following formula:

$\frac{\sqrt[]{ \sum _{i}^{} \left( X_{i}-Y_{i} \right) ^{2}}}{ \left( \sqrt[]{ \sum _{i}^{}X_{i}^{2}} \right) \times \left( \sqrt[]{ \sum _{i}^{}Y_{i}^{2}} \right) } \$

Where Xi represents the number of patents belonging to a specific class in year t, and Yi represents the number of patents belonging to a specific class in the previous year (t-1).

The difference here is that I want to add another assumption: If a year/some years is/are missing in between, I want to implement the assumption that the firm has been active in the same patent classes as the latest non-missing year. For example, in the following dataset:

> set.seed(123)
> df <- data.table(firm = rep(c("A","B"),each=5),
               year = c(1979,1981,1981,1984,1984,1959,1960,1963,1963,1965),
               patent = sample(3800000:4200000,10,replace = FALSE),
               class = c("410","73","52","250","252","105","454","380","380","60")
               )
> df
    firm year  patent class
 1:    A 1979 3988941   410
 2:    A 1981 3934057    73
 3:    A 1981 3924021    52
 4:    A 1984 3960996   250
 5:    A 1984 4026317   252
 6:    B 1959 4165208   105
 7:    B 1960 3924506   454
 8:    B 1963 3993626   380
 9:    B 1963 3845403   380
10:    B 1965 3865160    60

Firm A is missing the years 1980, 1982, and 1983. For the year 1980, the assumption is that firm has been active in the same technological patent as the year 1979: a single patent in the class 410. For the year 1982 and 1983, the assumption is that firm has been active in the same technological patent as the year 1981: two patents in classes 73 and 52 each. Therefore, the assumption is manifested in this way:

> df
    firm year  patent class
 1:    A 1979 4108578   410
 2:    A 1980 4108578   410
 3:    A 1981 3859133    73
 4:    A 1981 3983203    52
 5:    A 1982 3859133    73
 6:    A 1982 3983203    52
 7:    A 1983 3859133    73
 8:    A 1983 3983203    52
 9:    A 1984 4158992   250
10:    A 1984 3945254   252
11:    B 1959 4077323   105
12:    B 1960 3889708   454
13:    B 1961 3889708   454
14:    B 1962 3889708   454
15:    B 1963 3830537   380
16:    B 1963 4025588   380
17:    B 1964 3830537   380
18:    B 1964 4025588   380
19:    B 1965 3944607    60

Here again, for firm A, since year 1979 is the beginning year , there is no Euclidean distance for that year (NAs should be produced). Moving forward to year 1980, the distance is zero. For year 1981, the distinct classes for this year (1981) and the previous year (1980) are 73, 52, and 410. Therefore, the above formula is summed over these three distinct classes (there is three distinct 'i's). So the formula's output will be:

$\frac{\sqrt{(1-0)^{2}+(0-1)^{2}+(0-1)^{2}}}{\left(\sqrt{(1+0+0)^{2}}\right) \times\left(\sqrt{(0+1+1)^{2}}\right)}$

To add more clarification, the computation for year 1984 is explained: In year 1984, firm A has a total of two patents in classes 250 and 252 (one in each). The immediate previous year is 1983, which was originally missing, but after applying the above assumption, now it has two patents in classes 73 and 52 each. Because the distance is only between two consecutive years, to calculate the distance for year 1984, only years 1984 and 1983 are considered. Therefore, we have a total of four classes (250, 252, 73, and 52), meaning the summation is done over four 'i's. Beginning with the first i (class 250), the total number of patents in this class in 1 for 1984 and 0 for 1983, so Xi is equal to 1 and Yi is equal to 0. The same logic applies to 252 (Xi=1, and Yi=0). Now for the third 'i', or class 73, the total number of patents is 0 in 1984 and 1 in 1983, so Xi is equal to 0 and Yi is equal to 1. The same logic applies to class 52. Therefore the distance is given by:

$\frac{\sqrt{(1-0)^{2}+(1-0)^{2}+(0-1)^{2}+(0-1)^{2}}}{\sqrt{1^2+1^2+0^2+0^2}\times \sqrt{0^2+0^2+1^2+1^2}} = 1$

Following the same calculation and reiterating over firms, the final output should be:

> df
    firm year  patent class  El_Dist
 1:    A 1979 4108578   410       NA
 2:    A 1980 4108578   410 0.000000
 3:    A 1981 3859133    73 1.224745
 4:    A 1981 3983203    52 1.224745
 5:    A 1982 3859133    73 0.000000
 6:    A 1982 3983203    52 0.000000
 7:    A 1983 3859133    73 0.000000
 8:    A 1983 3983203    52 0.000000
 9:    A 1984 4158992   250 1.000000
10:    A 1984 3945254   252 1.000000
11:    B 1959 4077323   105       NA
12:    B 1960 3889708   454 1.414214
13:    B 1961 3889708   454 0.000000
14:    B 1962 3889708   454 0.000000
15:    B 1963 3830537   380 1.118034
16:    B 1963 4025588   380 1.118034
17:    B 1964 3830537   380 0.000000
18:    B 1964 4025588   380 0.000000
19:    B 1965 3944607    60 1.118034

I'm preferably looking for a data.table solution for speed purposes (my raw data contains about 7 million rows).

Thank you very much in advance for any help.

user_3732271 · Answer

The expansion can be done as

df1 <- df[, lapply(.SD, list), .(firm, year)][df[,
     .(year = min(year):max(year)), firm], on = .(firm, year)]
df1[, grp := cumsum(sapply(patent, Negate(is.null))), .(firm)]
df1[,  c('patent', 'class') := list(patent[1], class[1]), .(firm, grp)]
df1[, .(patent = unlist(patent), class = unlist(class)), .(firm, year)]

-output

#     firm year  patent class
# 1:    A 1979 3988941   410
# 2:    A 1980 3988941   410
# 3:    A 1981 3934057    73
# 4:    A 1981 3924021    52
# 5:    A 1982 3934057    73
# 6:    A 1982 3924021    52
# 7:    A 1983 3934057    73
# 8:    A 1983 3924021    52
# 9:    A 1984 3960996   250
#10:    A 1984 4026317   252
#11:    B 1959 4165208   105
#12:    B 1960 3924506   454
#13:    B 1961 3924506   454
#14:    B 1962 3924506   454
#15:    B 1963 3993626   380
#16:    B 1963 3845403   380
#17:    B 1964 3993626   380
#18:    B 1964 3845403   380
#19:    B 1965 3865160    60

user_3521006 · Answer

EDIT: update after some clarification

Here's a mostly (though not completely) vectorized approach with 'implicit expansion':

foo = function(x, y) {
  sqrt(sum((x - y)^2)) / (sqrt(sum(x^2)) * sqrt(sum(y^2)))
}

bar = function(x, y) {
  y = unlist(y, use.names = FALSE)
  vals = union(x, y)
  list(
    x = sapply(vals, function(v) sum(x == v)),
    y = sapply(vals, function(v) sum(y == v))
  )
}

x = df[, .(prev_class = list(class)), by = .(year, firm)]

df[x, 
   on = .(firm, year > year), 
   prev_class := i.prev_class]

df[, dist :=  {
  temp = bar(class, prev_class[1L])
  foo(temp$x, temp$y)
}, by = .(firm, year)]

df
#     firm year  patent class prev_class     dist
#  1:    A 1979 3988941   410                 Inf
#  2:    A 1981 3934057    73        410 1.224745
#  3:    A 1981 3924021    52        410 1.224745
#  4:    A 1984 3960996   250      73,52 1.000000
#  5:    A 1984 4026317   252      73,52 1.000000
#  6:    B 1959 4165208   105                 Inf
#  7:    B 1960 3924506   454        105 1.414214
#  8:    B 1963 3993626   380        454 1.118034
#  9:    B 1963 3845403   380        454 1.118034
# 10:    B 1965 3865160    60    380,380 1.118034

original answer:

To compute the distance with "implicit expansion" you can use the following approach. However, my results differ from OP's expected output for firm B in the years 1963 and 1965. It's unclear to me, how these results were computed by OP.

foo = function(x, y) {
  sqrt(sum((x - y)^2)) / (sqrt(sum(x^2)) * sqrt(sum(y^2)))
}

bar = function(x, y) {
  y = unlist(y, use.names = FALSE)
  vals = union(x, y)
  list(
    x = as.integer(vals %in% x), 
    y = as.integer(vals %in% y)
  )
}

x = df[, .(prev_class = list(unique(class))), by = .(year, firm)]

df[x, 
   on = .(firm, year > year), 
   prev_class := i.prev_class]

df[, dist :=  {
  temp = bar(class, prev_class)
  foo(temp$x, temp$y)
}, by = .(firm, year)]

df
#     firm year  patent class prev_class     dist op_expected
#  1:    A 1979 3988941   410                 Inf          NA
#  2:    A 1981 3934057    73        410 1.224745    1.224745
#  3:    A 1981 3924021    52        410 1.224745    1.224745
#  4:    A 1984 3960996   250      73,52 1.000000    1.000000
#  5:    A 1984 4026317   252      73,52 1.000000    1.000000
#  6:    B 1959 4165208   105                 Inf          NA
#  7:    B 1960 3924506   454        105 1.414214    1.414214
#  8:    B 1963 3993626   380        454 1.414214    1.118034
#  9:    B 1963 3845403   380        454 1.414214    1.118034
# 10:    B 1965 3865160    60        380 1.414214    1.118034

user_11740629 · Answer

Not an answer, but an extension to the answer provided by talat.

Here's what prevents my modification to be the final answer:

I'm sure there's a simpler and more elegant way to do what I've done.
The answer correctly implements the assumption, but does not directly give outputs for the missing years. Sure this expansion by akrun can be applied first, but maybe there's a way to integrate both steps into one.

Anyways, here's my modification that works according to the formula:

foo = function(x, y) {
  sqrt(sum((x - y)^2)) / (sqrt(sum(x^2)) * sqrt(sum(y^2)))
  }

bar = function(x, y) {
  x = unlist(x, use.names = FALSE)
  y = unlist(y, use.names = FALSE)
  vals = c(x, y)
  xl = length(unique(x))
  yl = length(unique(y))
  ul = length(union(x,y))
  list(
    x = c(table(vals)[names(table(vals)) %in% x],rep(0,(ul-xl))), 
    y = c(rep(0,(ul-yl)),table(vals)[names(table(vals)) %in% y])
  )
  }

x1 = df[, .(prev_class = list(class)), by = .(year, firm)]
x2 = df[, .(curr_class = list(class)), by = .(year, firm)]
x1
x2
x = merge(x1,x2)

df[x, 
   on = .(firm, year > year), 
   prev_class := i.prev_class]

df[x, 
   on = .(firm, year), 
   curr_class := curr_class]

df[, dist :=  {
  temp = bar(unique(curr_class), unique(prev_class))
  foo(temp$x, temp$y)
  }, by = .(firm, year)]

df
    firm year  patent class prev_class curr_class     dist
 1:    A 1979 3988941   410                   410      Inf
 2:    A 1981 3934057    73        410      73,52 1.224745
 3:    A 1981 3924021    52        410      73,52 1.224745
 4:    A 1984 3960996   250      73,52    250,252 1.000000
 5:    A 1984 4026317   252      73,52    250,252 1.000000
 6:    B 1959 4165208   105                   105      Inf
 7:    B 1960 3924506   454        105        454 1.414214
 8:    B 1963 3993626   380        454    380,380 1.118034
 9:    B 1963 3845403   380        454    380,380 1.118034
10:    B 1965 3865160    60    380,380         60 1.118034

Euclidean distant for NON-CONSECUTIVE classes of factors iterated by groups

3 Answers