R httr download files from ftp error 421 "too many connections from your internet address"

Viewed 1118

EDIT - Short question: Does httr have a finalizer that closes the FTP connection?

I'm downloading climate projections files from the ftp server of the NASA NEX project using the httr package.

My script is:

library(httr)

var = c("pr", "tasmin", "tasmax")
rcp = c("rcp45", "rcp85")
mod= c("inmcm4", "GFDL-CM3")
year=c(seq(2040,2080,1))

for (v in var) {
  for (r in rcp) {
    url<- paste0( 'ftp://ftp.nccs.nasa.gov/BCSD/', r, '/day/atmos/', v, '/r1i1p1/v1.0/', sep='')
    for (m in mod) {
  for (y in year) {
    nfile<- paste0(v,'_day_BCSD_',r,"_r1i1p1_",m,'_',y,'.nc', sep='')
    url1<- paste0(url,nfile, sep='')
    destfile<-paste0('mypath',r,'/',v,'/',nfile, sep='')
    GET(url=url1, authenticate(user='NEXGDDP', password='', type = "basic"), write_disk(path=destfile, overwrite = FALSE ))
    Sys.sleep(0.5)
  }}}}

After a while, the server stops my connection with the following error: "421 There are too many connections from your internet address".

I read here that this is due to the number of connections open and that I should close them at each iteration (I'm not sure this does really make sense tho!). Is there a way to close the ftp with the httr package?

2 Answers
Related