loop and apply family functions for climate and yield data to calculate correlation

user_19125895 0
Viewed 70

I have a file including around 350 columns; year, temperature for each day , yield for different sites. I need to group or split data by year, then calculate the correlation test between yield and each temperature column one by one. I wrote the script below, however, it produce the results only for one year, is there any suggestion where is the problem/issue (it does not go through each year).

for (Y in unique(data_final$YEAR)) {
  # cat ("\n\n YEAR =", Y, "\n =========") # Write year Number
  subData <- data_final [data_final$YEAR == Y,] # Subset the data
  Tmax <- subData[, grepl ("TMAX", colnames (subData))]
  Yield <- subData$YIELD # get YIELD column
  cortest <- list ()
  
  for (i in 1:length (Tmax)) {
  cortest[[i]] <- cor(Tmax[[i]], Yield, use="pairwise.complete.obs", method = "pearson")
  
  }
  return(do.call ("rbind", cortest))
 }
sapply r loops split
3 Answers

Here is the answer

corrresults <- as.data.frame(unique(data_final$YEAR))
Tmax <- data_final[, grepl ("TMAX", colnames (data_final))]
datasetup <- as.data.frame(matrix(data = NA, nrow=length(YEAR), ncol = length(Tmax)))
corrresults <- cbind(corrresults, datasetup)
colnames(corrresults) <- c("YEAR", seq(1, length(Tmax)))

for (Y in 1:length(YEAR)) {
  
  subData <- data_final[data_final$YEAR == YEAR[Y],] # Subset the data
   
  Tmax <- subData[, grepl ("TMAX", colnames (subData))]
  Yield <- subData$YIELD # get YIELD column
  
  for (i in 1:length (Tmax)) {# Iterate over columns start with Tmax
    cortest <- cor(Tmax[[i]], Yield, use="pairwise.complete.obs", method = "pearson")
    corrresults[[Y, i+1]] <- cortest
      
  } # end of loop for 
 
} # end of loop for YEAR

write.csv(corrresults, file = "corrresults.csv")
user_191258950

Sounds like a split, apply, combine task to me. So maybe:

sp <- split(data_final, data_final$YEAR)
one_year <- function(dset) {
    message("=== year: ", dset[1,"YEAR"], "===")
    # your code
}
res_list <- lapply(sp, one_year)
res <- do.call(rbind, res_list)

can do the trick.

The problem with your code seems to be that you use return in the outer for loop. You would want to collect cortest somehow and then enter the next iteration of the loop.

user_2160640

If you are looking for a matrix of correlation between Temp and Yield for Years in your data, you can simply use this functionality of tidyverse and tidymodels.

## Load libraries

library(tidyverse)
library(tidymodels)


## Load data

data_final <- read.csv("Downloads/data_final_winner.csv")

## Correlation

data_final |> 
  select(-c(1, 348:351, 353)) |> 
  pivot_longer(names_to = "Temp", values_to = "value", cols = 2:346) |> 
  group_by(YEAR, Temp) %>%
  summarize(correlation = cor(YIELD, value)) |> 
  pivot_wider(names_from = Temp, values_from = correlation)

It should give you the output you are looking.

Created on 2022-08-26 with reprex v2.0.2

user_113439630
Related