how to use download.file to download images from web in r

Viewed 27

I have the following code (edited):

library(rvest)
library(httr)
folder <- "~/Downloads/DSA2101 Main/data"
dir.create(file.path(folder, "radar_files"))
interval <- seq(0, 55, by = 5)
hour <- c("01", "02", "03")
for (h in hour) {
  for (i in interval) {
    print(i, length(i))
    if (i == 0 | i == 5) {
      url <- paste("https://www.nea.gov.sg/docs/default-source/rain-area-240km/dpsri_240km_20220731", h, "0", i, "0000dBR.dpsri.png", sep = "")
    } else {
       url <- paste("https://www.nea.gov.sg/docs/default-source/rain-area-240km/dpsri_240km_20220731", h, i, "0000dBR.dpsri.png", sep = "")
    }
    download.file (imgurl, destfile = file.path(folder, "radar_files", basename(url)))
  }
} 

I am trying to webscrape the 240km radar scans from the aforementioned url in the code and download them into radar_files before zipping the file. Radar_files is a folder I created. However, when I run the code, I get (edited)

trying URL 'NA' Warning: URL 'NA': status was 'Couldn't resolve host name'Error in download.file(imgurl, destfile = file.path(folder, "radar_files", : cannot open URL 'NA'

Where did I go wrong?

Thank you.

1 Answers
  1. From the error message I would guess that the folder does not exist.

  2. Even if the folder exists your destfile contains only the path to the folder. You also have to add a filename. To this end I use basename(url) which returns the original filename from the url.

  3. As your url already contains the url to the png I don't understand what you are trying to achieve with the rvest code.

This said, below is a minimal reprex which works fine for me.

Note: I download only three of the images.

# Create a temporary directory for the reprex
folder <- tempdir() # Replace with the path to your folder

# Create folder to the store the pngs
dir.create(file.path(folder, "radar_files"))

interval <- c(0, 5, 10)
hour <- c("01")

for (h in hour) {
  for (i in interval) {
    print(i, length(i))
    if (i %in% c(0, 5)) {
      url <- paste("https://www.nea.gov.sg/docs/default-source/rain-area-240km/dpsri_240km_20220731", h, "0", i, "0000dBR.dpsri.png", sep = "")
    } else {
      url <- paste("https://www.nea.gov.sg/docs/default-source/rain-area-240km/dpsri_240km_20220731", h, i, "0000dBR.dpsri.png", sep = "")
    }
    
    download.file(url, destfile = file.path(folder, "radar_files", basename(url)))
  }
}
#> [1] 0
#> [1] 5
#> [1] 10

dir(file.path(folder, "radar_files"))
#> [1] "dpsri_240km_2022073101000000dBR.dpsri.png"
#> [2] "dpsri_240km_2022073101050000dBR.dpsri.png"
#> [3] "dpsri_240km_2022073101100000dBR.dpsri.png"

UPDATE Of course could we get rid of the for loops and use e.g. map ply. To this end I first create a dataframe containing the urls of the pngs to download and the file names. Afterwards we can loop over both the url and the destfile. Additionally I simplified your code a bit by using paste0 and strings::str_pad. Finally, unfortunatly I can' reproduce your error. The only issue with your edited code is that you use imgurl while in my code it was url. But that gives me a different error. To check for any errors related to the fact that I only downloaded three of the pngs I extended the reprex to include all intervals and hours:

# Create a temporary directory for the reprex
folder <- tempdir() # Replace with the path to your folder

# Create folder to the store the pngs
dir.create(file.path(folder, "radar_files"))

urls <- expand.grid(
  interval = seq(0, 55, by = 5),
  hour = c("01", "02", "03")
)
urls$interval <- stringr::str_pad(urls$interval, width = 2, pad = 0)
urls$url <- paste0("https://www.nea.gov.sg/docs/default-source/rain-area-240km/dpsri_240km_20220731", urls$hour, urls$interval, "0000dBR.dpsri.png")
urls$destfile <- file.path(folder, "radar_files", basename(urls$url))

foo <- mapply(function(x, y) download.file(x, y), urls$url, urls$destfile)

dir(file.path(folder, "radar_files"))
#>  [1] "dpsri_240km_2022073101000000dBR.dpsri.png"
#>  [2] "dpsri_240km_2022073101050000dBR.dpsri.png"
#>  [3] "dpsri_240km_2022073101100000dBR.dpsri.png"
#>  [4] "dpsri_240km_2022073101150000dBR.dpsri.png"
#>  [5] "dpsri_240km_2022073101200000dBR.dpsri.png"
#>  [6] "dpsri_240km_2022073101250000dBR.dpsri.png"
#>  [7] "dpsri_240km_2022073101300000dBR.dpsri.png"
#>  [8] "dpsri_240km_2022073101350000dBR.dpsri.png"
#>  [9] "dpsri_240km_2022073101400000dBR.dpsri.png"
#> [10] "dpsri_240km_2022073101450000dBR.dpsri.png"
#> [11] "dpsri_240km_2022073101500000dBR.dpsri.png"
#> [12] "dpsri_240km_2022073101550000dBR.dpsri.png"
#> [13] "dpsri_240km_2022073102000000dBR.dpsri.png"
#> [14] "dpsri_240km_2022073102050000dBR.dpsri.png"
#> [15] "dpsri_240km_2022073102100000dBR.dpsri.png"
#> [16] "dpsri_240km_2022073102150000dBR.dpsri.png"
#> [17] "dpsri_240km_2022073102200000dBR.dpsri.png"
#> [18] "dpsri_240km_2022073102250000dBR.dpsri.png"
#> [19] "dpsri_240km_2022073102300000dBR.dpsri.png"
#> [20] "dpsri_240km_2022073102350000dBR.dpsri.png"
#> [21] "dpsri_240km_2022073102400000dBR.dpsri.png"
#> [22] "dpsri_240km_2022073102450000dBR.dpsri.png"
#> [23] "dpsri_240km_2022073102500000dBR.dpsri.png"
#> [24] "dpsri_240km_2022073102550000dBR.dpsri.png"
#> [25] "dpsri_240km_2022073103000000dBR.dpsri.png"
#> [26] "dpsri_240km_2022073103050000dBR.dpsri.png"
#> [27] "dpsri_240km_2022073103100000dBR.dpsri.png"
#> [28] "dpsri_240km_2022073103150000dBR.dpsri.png"
#> [29] "dpsri_240km_2022073103200000dBR.dpsri.png"
#> [30] "dpsri_240km_2022073103250000dBR.dpsri.png"
#> [31] "dpsri_240km_2022073103300000dBR.dpsri.png"
#> [32] "dpsri_240km_2022073103350000dBR.dpsri.png"
#> [33] "dpsri_240km_2022073103400000dBR.dpsri.png"
#> [34] "dpsri_240km_2022073103450000dBR.dpsri.png"
#> [35] "dpsri_240km_2022073103500000dBR.dpsri.png"
#> [36] "dpsri_240km_2022073103550000dBR.dpsri.png"
Related