Nest models dplyr calibration and validation

user_2950078 0
Viewed 49

I have used R quite a bit, but I'm starting my journey in the tidyverse. I'm trying to create a function that allows me to Bias correction daily precipitation series. I want to break the time series in 2 (for calibration and validation). I would need to fit the model for the calibration period, apply it to the validation period, together with the observed and modeled data. So far, I was able to do this in two for loops, but i was wondering if would be possible to do this "tidyer", with nest, but i cant figure it out.

Moreover, how could I use apply to compute this to many precipitation time series in a data.frame.

My current code is below,

Thanks in advance!

libraries
library(lubridate)
library(qmap)
library(dplyr)
Simulate data
obs_ <- runif(min=0,max=157,n=14975) 
sim <- obs_ + 20
date_ <- seq(as.Date("1979-01-01"), as.Date("2019-12-31"),by="days")
db <- data.frame(obs=obs_, sim=sim_, date=date_, month=month(date_), year=year(date_))
Sample years
ss<- seq(from=1979, to=2019, by=1)
samp <- sample(ss, length(ss)/2)
samp <- samp[order(samp)]
samp1 <- subset(ss, !(ss %in% samp))
Model
list_mod <- list()
for(i in 1:12){
  
  #  retrives the data for the calibration period
  model_fit <-db  %>% 
    mutate(id = case_when( year %in% samp ~ "cal",
                           year %in% samp1 ~ "val")) %>%
    filter(month== i, id== "cal")
 # fits the model to each month and stores it in a list
  list_mod[[i]] <- fitQmap(model_fit$obs,model_fit$sim)
 }
Retrives the data for the validation period
model1 <- db %>% 
  mutate(id = case_when( year %in% samp ~ "cal",
                         year %in% samp1 ~ "val")) %>%
  filter(id=="val")
Estimates the new data and stores it with the observations and simulations
for( i in 1:12){
 
  temp__ <-  model1[model1$month ==i,"sim"]
  model1[model1$month ==i,"model"] <-  doQmap(temp__, list_mod[[i]])
  
  
}
dplyr r
1 Answers

If you're not wedded to tidy, here is a solution using data.table.

Using your db:

library(data.table)
library(qmap)
##
setDT(db)[, Set:='cal']
db[sample(.N, .N/2), Set:='val']
db[, pred:=doQmap(sim, fitQmap(obs[Set=='cal'], sim[Set=='cal'])), by=.(month)]
result <- db[Set=='val']

The first line converts your db to a data.table and creates a column, Set, to define calibration/validation. The second line assigns a random 1/2 of the data to the validation set.

The third line does all the work: it groups the rows by month ( by=.(month) ), then generates fits with fitQmap(...) on the calibration set, and then generates debiased predictions using doQmap(...) on the full dataset.

The final line just filters out the calibration rows.

I notice in this example that Qmap reduces but does not eliminate bias. It that what you expect?

user_29850070
Related