Why does .SD in a data.table non-equi join throws an error occasionally?

Viewed 104

I have noticed an inconsistency when using .SD in a non-equi join. Is there an explanation for this?

Depending on the "direction" or "type" of join, using j = .SD throws an error.

library(data.table)
d1 <- fread("a, b
            1, 11
            6, 16")
d2 <- data.table(r = 1:5, s = seq(0, 20, 5))

d1
   a  b
1: 1 11
2: 6 16
d2
   r  s
1: 1  0
2: 2  5
3: 3 10
4: 4 15
5: 5 20
d1[d2, on = .(a <= s, b >= s)]
    a  b r
1:  0  0 1
2:  5  5 2
3: 10 10 3
4: 10 10 3
5: 15 15 4
6: 20 20 5
d1[d2, on = .(a <= s, b >= s), j = .SD]

Error in [.data.table(d1, d2, on = .(a <= s, b >= s), j = .SD) :
column(s) not found: a

d2[d1, on = .(s >= a, s <= b)]
   r s s.1
1: 2 1  11
2: 3 1  11
3: 3 6  16
4: 4 6  16
d2[d1, on = .(s >= a, s <= b), j = .SD]
   r s
1: 2 1
2: 3 1
3: 3 6
4: 4 6

I have reproduced the behaviour with R version 3.6.0 and data.table versions 1.11.8, 1.12.2, and 1.12.3 (development version on github).

I am aware that there are related discussions on github, e.g., Both columns for rolling and non-equi joins #3093, .SD in expression with j? #3115 but I haven't found (perhaps overlooked?) an explanation for the observed behaviour there.

1 Answers

Thanks to Arun and Matt, the issue has been fixed (note item 24) with the latest development version of data.table 1.12.3.

So,

d1[d2, on = .(a <= s, b >= s), j = .SD]

no longer throws an error but returns

    a  b
1:  0  0
2:  5  5
3: 10 10
4: 10 10
5: 15 15
6: 20 20

as expected.

Related