How do I to skip to next site in for loop if missing value where TRUE/FALSE needed?

Viewed 45

I am trying to pull out all United States Geological Survey (USGS) sites that have daily mean water temperature for a given date range using the USGS package dataRetrieval. I can do this with a for loop that iterates through each US state/territory abbreviation but the loop stops running once it comes across a location that does not have any sites. How can I make the loop skip locations that do no have sites available?

This is the for loop that works but stops when it comes across a state/territory that does not have a site.

library(dataRetrieval)

statCd <- "00003" # Mean
startDate <- "1990-01-01"
endDate <- "2019-12-31"
states <- stateCd

# For loop determines how many sites have daily avg. water temp. data records for period of interest 
for(i in 1:length(states)) {
  tempDat_states = whatNWISdata(stateCd = states[i],
                                parameterCd = "00010") # Water temperature
  temp_states = tempDat_states %>%
    filter(stat_cd == "00003") %>%
    mutate(period = as.Date(endDate) - as.Date(startDate))
  temp_states$end_year <- year(temp_states$end_date)
  temp_states$start_year <- year(temp_states$begin_date)
  temp_states <- temp_states[temp_states$start_year <= 1990 & temp_states$end_year >= 2019,]
  curDat_temp = temp_states
  curDat_temp$STUSAB = states[i]
  if( i == 1){
    saveDat_temp = curDat_temp
  } else{
    saveDat_temp = rbind(saveDat_temp, curDat_temp)
  }
}

The for loop below is the same as above, but I have manually kept only the states/territories that have sites available. I would like to avoid having to manually identify and keep those states/territories, so any code that skips those locations (maybe using next?) in the for loop would be what I'm looking for.

# Remove AR, DO, HI, ID, IA, MA, NE, NH, OK, RI, SD, VT, AS, GU, MP, PR, UM, VI for having zero sites
states <- stateCd[c(1:3,5:8,10:11,14:15,17:21,23:27,29,31:36,38:39,41,43:45,47:51),2]

for(i in 1:length(states)) {
  tempDat_states = whatNWISdata(stateCd = states[i],
                                parameterCd = "00010") # Water temperature
  temp_states = tempDat_states %>%
    filter(stat_cd == "00003") %>%
    mutate(period = as.Date(endDate) - as.Date(startDate))
  temp_states$end_year <- year(temp_states$end_date)
  temp_states$start_year <- year(temp_states$begin_date)
  temp_states <- temp_states[temp_states$start_year <= 1990 & temp_states$end_year >= 2019,]
  curDat_temp = temp_states
  curDat_temp$STUSAB = states[i]
  if( i == 1){
    saveDat_temp = curDat_temp
  } else{
    saveDat_temp = rbind(saveDat_temp, curDat_temp)
  }
}
2 Answers

Don't grow objects in for loop. Define a list and fill it in each iteration. Try :

library(dataRetrieval)
library(dplyr)

statCd <- "00003" 
startDate <- "1990-01-01"
endDate <- "2019-12-31"
states <- stateCd
result <- vector('list', nrow(states))

for(i in 1:nrow(states)) {
  print(i)
  tempDat_states = whatNWISdata(stateCd = states$STUSAB[i],parameterCd = "00010")
  temp_states = tempDat_states %>%
    filter(stat_cd == "00003") %>%
    mutate(period = as.Date(endDate) - as.Date(startDate))
  temp_states$end_year <- year(temp_states$end_date)
  temp_states$start_year <- year(temp_states$begin_date)
  temp_states <- temp_states[temp_states$start_year <= 1990 & temp_states$end_year >= 2019,]
  result[[i]] <- temp_states
}

full_data <- do.call(rbind.data.frame, result)

Here is an approach which uses lapply() instead of a for loop.

library(dataRetrieval)
library(dplyr)
library(magrittr)
statCd <- "00003" # Mean
startDate <- "1990-01-01"
endDate <- "2019-12-31"
states <- setdiff(stateCd$STUSAB, "UM")
saveDat <- lapply(
  states,
  function(st) {
    cat("Retrieve data for:", st, "\n")
    whatNWISdata(
      stateCd = st,
      parameterCd = "00010" # Water temperature
    ) %>%
      filter(stat_cd == statCd & begin_date >= startDate & end_date <= endDate) %T>%
      {cat(st, ":", nrow(.), "rows after filtering\n")}
  }) %>%
  set_names(states)

returns a named list with the data from each state (one data.frame per list element).

As the download takes quite a while, the calls to cat() inform about the progress and the success for each state:

Retrieve data for: AL 
AL : 1 rows after filtering
Retrieve data for: AK 
|========================================================================================| 100% 27 MB
AK : 47 rows after filtering
Retrieve data for: AZ 
|========================================================================================| 100% 60 MB
|========================================================================================| 100% 60 MB
AZ : 3 rows after filtering
Retrieve data for: AR 
|========================================================================================| 100% 34 MB
AR : 8 rows after filtering
Retrieve data for: CA 
|=======================================================================================| 100% 264 MB
|=======================================================================================| 100% 264 MB
|=======================================================================================| 100% 264 MB
CA : 20 rows after filtering
Retrieve data for: CO 
CO : 19 rows after filtering
Retrieve data for: CT 
|========================================================================================| 100% 23 MB
CT : 18 rows after filtering
Retrieve data for: DE 
DE : 4 rows after filtering
Retrieve data for: DC 
DC : 6 rows after filtering
Retrieve data for: FL 
|=======================================================================================| 100% 107 MB
FL : 76 rows after filtering
Retrieve data for: GA 
GA : 39 rows after filtering
Retrieve data for: HI 
HI : 99 rows after filtering
Retrieve data for: ID 
|========================================================================================| 100% 83 MB
ID : 76 rows after filtering
Retrieve data for: IL 
IL : 25 rows after filtering
Retrieve data for: IN 
|========================================================================================| 100% 44 MB
IN : 9 rows after filtering
Retrieve data for: IA 
|========================================================================================| 100% 70 MB
IA : 17 rows after filtering
Retrieve data for: KS 
|========================================================================================| 100% 67 MB
KS : 47 rows after filtering
Retrieve data for: KY 
KY : 32 rows after filtering
Retrieve data for: LA 
|========================================================================================| 100% 53 MB
LA : 29 rows after filtering
Retrieve data for: ME 
ME : 16 rows after filtering
Retrieve data for: MD 
|========================================================================================| 100% 43 MB
MD : 22 rows after filtering
Retrieve data for: MA 
MA : 4 rows after filtering
Retrieve data for: MI 
|========================================================================================| 100% 42 MB
MI : 41 rows after filtering
Retrieve data for: MN 
MN : 147 rows after filtering
Retrieve data for: MS 
MS : 9 rows after filtering
Retrieve data for: MO 
MO : 21 rows after filtering
Retrieve data for: MT 
MT : 38 rows after filtering
Retrieve data for: NE 
NE : 32 rows after filtering
Retrieve data for: NV 
|========================================================================================| 100% 50 MB
NV : 41 rows after filtering
Retrieve data for: NH 
NH : 4 rows after filtering
Retrieve data for: NJ 
|========================================================================================| 100% 93 MB
NJ : 68 rows after filtering
Retrieve data for: NM 
|========================================================================================| 100% 59 MB
|========================================================================================| 100% 59 MB
NM : 11 rows after filtering
Retrieve data for: NY 
|=======================================================================================| 100% 128 MB
|=======================================================================================| 100% 128 MB
NY : 34 rows after filtering
Retrieve data for: NC 
|========================================================================================| 100% 47 MB
NC : 108 rows after filtering
Retrieve data for: ND 
ND : 16 rows after filtering
Retrieve data for: OH 
OH : 10 rows after filtering
Retrieve data for: OK 
OK : 26 rows after filtering
Retrieve data for: OR 
|========================================================================================| 100% 32 MB
OR : 122 rows after filtering
Retrieve data for: PA 
|=======================================================================================| 100% 108 MB
PA : 91 rows after filtering
Retrieve data for: RI 
RI : 0 rows after filtering
Retrieve data for: SC 
SC : 47 rows after filtering
Retrieve data for: SD 
SD : 10 rows after filtering
Retrieve data for: TN 
TN : 16 rows after filtering
Retrieve data for: TX 
|========================================================================================| 100% 99 MB
|========================================================================================| 100% 99 MB
TX : 120 rows after filtering
Retrieve data for: UT 
UT : 15 rows after filtering
Retrieve data for: VT 
VT : 3 rows after filtering
Retrieve data for: VA 
|========================================================================================| 100% 34 MB
VA : 86 rows after filtering
Retrieve data for: WA 
|========================================================================================| 100% 84 MB
WA : 57 rows after filtering
Retrieve data for: WV 
WV : 13 rows after filtering
Retrieve data for: WI 
WI : 70 rows after filtering
Retrieve data for: WY 
|========================================================================================| 100% 53 MB
WY : 24 rows after filtering
Retrieve data for: AS 
AS : 0 rows after filtering
Retrieve data for: GU 
GU : 0 rows after filtering
Retrieve data for: MP 
MP : 0 rows after filtering
Retrieve data for: PR 
PR : 0 rows after filtering
Retrieve data for: VI 
VI : 0 rows after filtering

UM was removed from the vector of state abbreviations because whatNWISdata() runs into an error when call with stateCd = "UM".

saveDat is a large list of 56 elements but the OP expects one data.frame with the combined results.

Unfortunately, the data.frames returned by whatNWISdata() are not always consistent. This is why row_bind() fails because it carefully checks that the type of columns do match:

bind_rows(saveDat, .id = "STUSAB")
Error: Can't combine `AL$alt_acy_va` <character> and `IL$alt_acy_va` <double>.

do.call(rbind, saveDat) is less strict and binds the data.frames without issuing any warning.

Related