How would you lookup the next-closest value?

Viewed 187

I have the following 2 data.frames:

data.df <- data.frame(dt = as.POSIXct(c('2020-01-08 11:30:00', 
    '2020-01-10 11:30:00', '2020-01-11 12:30:00')), 
  v1=c(1,2,3))

lookup.df <- data.frame(ldt = as.POSIXct(c('2020-01-08 11:29:00', 
  '2020-01-08 11:30:00', '2020-01-08 11:31:00', '2020-01-10 10:30:00', 
  '2020-01-10 11:31:00', '2020-01-11 11:30:00', '2020-01-12 11:30:00')), 
   lv = 1:7)

For each row in data.df I would like to get a row index in lookup.df (to match rows in a merge) where lookup.df$ldt >= data.df$dt on the same day. If no date meets that requirement then NA. So in this example the ideal output would be:

dt                    |   v1   |  ldt                 |  lv
2020-01-08 11:30:00        1      2020-01-08 11:30:00     2
2020-01-10 11:30:00        2      2020-01-10 11:31:00     5
2020-01-11 12:30:00        3       NA                     NA

NOTE: I would prefer a base R implementation or a zoo implementation

4 Answers

1) Base R - sapply This uses base R. For each component of dt in data.df it finds all datetimes greater than it in lookup.df on the same date and then returns the index of the first. Finally it puts together data.df and the rows of those indexes of lookup.df .

ix <- sapply(data.df$dt, function(dt) with(lookup.df, 
  which(ldt >= dt & as.Date(ldt, tz = "") == as.Date(dt, tz = ""))[1]
))
res <- cbind(data.df, lookup.df[ix, ])
rownames(res) <- NULL

giving:

> res
                   dt v1                 ldt lv
1 2020-01-08 11:30:00  1 2020-01-08 11:30:00  2
2 2020-01-10 11:30:00  2 2020-01-10 11:31:00  5
3 2020-01-11 12:30:00  3                <NA> NA

2) Base R - merge This is an alternate base R approach. Append a date column to each input data frame and then merge the two by that column. Remove any row for which the lookup.df date/time is less than the data.df date/time and then take the first row of each set of rows derived from the same original data.df row. That will get the matches except that it will miss the rows that have no matches at all so perform a second merge to get those back.

data.df$date <- as.Date(data.df$dt, tz = "")
lookup.df$date <- as.Date(lookup.df$ldt, tz = "")

m <- merge(data.df, lookup.df, by = "date", all.x = TRUE, all.y = FALSE)
m <- subset(m, dt <= ldt)
m <- m[!duplicated(m[1:3]), ]
merge(data.df[-3], m[-1], by = c("dt", "v1"), all.x = TRUE, all.y = FALSE)

giving:

                   dt v1                 ldt lv
1 2020-01-08 11:30:00  1 2020-01-08 11:30:00  2
2 2020-01-10 11:30:00  2 2020-01-10 11:31:00  5
3 2020-01-11 12:30:00  3                <NA> NA

3) SQL Although the question asked for a base R solution an sql solution has been added in addition here because it provides a particularly straight forward translation of the problem into code as a self join with a complex condition. It performs a left join on the indicated condition and takes the minimum ldt found over all rows derived from the same row in data.df.

library(sqldf)

data.df$date <- as.Date(data.df$dt, tz = "")
lookup.df$date <- as.Date(lookup.df$ldt, tz = "")

sqldf("select D.dt, D.v1, min(L.ldt) as ldt, L.lv
  from [data.df] D left join [lookup.df] L
  on D.dt <= L.ldt and D.date == L.date
  group by D.rowid")

giving:

                   dt v1                 ldt lv
1 2020-01-08 11:30:00  1 2020-01-08 11:30:00  2
2 2020-01-10 11:30:00  2 2020-01-10 11:31:00  5
3 2020-01-11 12:30:00  3                <NA> NA

Note

There was a problem in the question with fancy quotes that R cannot read so we used this as the input:

data.df <- data.frame(dt = as.POSIXct(c('2020-01-08 11:30:00', 
    '2020-01-10 11:30:00', '2020-01-11 12:30:00')), 
  v1=c(1,2,3))

lookup.df <- data.frame(ldt = as.POSIXct(c('2020-01-08 11:29:00', 
  '2020-01-08 11:30:00', '2020-01-08 11:31:00', '2020-01-10 10:30:00', 
  '2020-01-10 11:31:00', '2020-01-11 11:30:00', '2020-01-12 11:30:00')), 
   lv = 1:7)

For the sake of completeness, here is a solution which uses data.table's rolling join.

If I understand correctly, the OP is looking for matches

  1. on the same day and
  2. on the first time stamp encountered in lookup.df on or after the time stamp given in `data.df

The second condition can be achieved by a simple rolling join:

library(data.table)
setDT(lookup.df)[setDT(data.df), on = .(ldt = dt), .(dt, v1, ldt = x.ldt, lv), roll = -Inf]
                    dt v1                 ldt lv
1: 2020-01-08 11:30:00  1 2020-01-08 11:30:00  2
2: 2020-01-10 11:30:00  2 2020-01-10 11:31:00  5
3: 2020-01-11 12:30:00  3 2020-01-12 11:30:00  7

However, it is obvious that the first condition is violated for row 3. In order to fulfill the first condition, we have to match on the same day as well. This requires to add a day column of type Date to both data frames:

library(data.table)
setDT(lookup.df)[, .(ldt, lv, day = as.IDate(ldt))][
  setDT(data.df)[, .(dt, v1, day = as.IDate(dt))], 
  on = .(day, ldt = dt), .(dt, v1, ldt = x.ldt, lv), roll = -Inf]
                    dt v1                 ldt lv
1: 2020-01-08 11:30:00  1 2020-01-08 11:30:00  2
2: 2020-01-10 11:30:00  2 2020-01-10 11:31:00  5
3: 2020-01-11 12:30:00  3                <NA> NA

Note that data.df and lookup.df aren't modified.

Assuming your lookup times are ordered, in base R you can do:

lv <- sapply(data.df$dt, function(x){
  which(substr(lookup.df$ldt, 1, 10) == substr(x, 1, 10) & lookup.df$ldt >= x)[1]
})

cbind(data.df, lookup.df[lv,])
#>                     dt v1                 ldt lv
#> 2  2020-01-08 11:30:00  1 2020-01-08 11:30:00  2
#> 5  2020-01-10 11:30:00  2 2020-01-10 11:31:00  5
#> NA 2020-01-11 12:30:00  3                <NA> NA

If you don't mind using lubridate you can use date() instead of substr()

And for the sake of complete and utter completeness, here's a dplyr version with a fuzzyjoin flavour:

library(fuzzyjoin)
library(dplyr)

fuzzy_left_join(data.df, lookup.df, by = c("day" = "day", "dt" = "ldt"), 

                match_fun = list(`==`, `<=`)) %>%
    select(-c(day.x, day.y)) %>%
    group_by(v1) %>% slice(1)

  dt                     v1 ldt                    lv
  <dttm>              <dbl> <dttm>              <int>
1 2020-01-08 11:30:00     1 2020-01-08 11:30:00     2
2 2020-01-10 11:30:00     2 2020-01-10 11:31:00     5
3 2020-01-11 12:30:00     3 NA                     NA
Related