How can I detect newly added files in a distant folder over the web in R?

I know the OP was looking for an "event listener" to monitor for changes on the file server, but some message has to be sent from the distant computer to notify your computer of the change. If you have no control over the file server, the only way to get it to send you a message is by first sending it a request. This means the only general "event listener" available is one that works by intermittently polling the server.

Depending on how frequently you poll, this should work perfectly well as an event listener. As an analogy, many species of bats hunt by sending out intermittent pulses of ultrasound and listening for the response. This is a form of intermittent polling that works well enough to keep them alive.

This does mean having to have some sort of software running in the background on your own computer. Your two options here are to use scheduling to run the R script intermittently, or to run an R script in the background that loops with a pause in between polls.

It appears from comments that the OP only wants to download any new files added to the server, but not to create copies of existing files when the program is first run. This means that a file has to be stored locally listing the contents of the ftp directory the last time it was checked, then comparing this to the current contents of the ftp directory, and downloading any new files as well as updating the content record.

Here's a function that does just that. The first time you run it, it will create a new local directory named after the hosting url and a .csv file with a listing of the directory at that point. Subsequent calls to the function after this will compare the contents of the local and remote directories and download any new files:

local_mirror <- function(url, root_dir = path.expand("~/"), silent = FALSE)
{
  if(substring(root_dir, nchar(root_dir), nchar(root_dir)) != "/")
    root_dir <- paste0(root_dir, "/")
  content <- rvest::html_nodes(xml2::read_html(url), "a")
  links <- rvest::html_attr(content, "href")
  links <- grep("/", links, invert = TRUE, value = TRUE)
  rel_path <- strsplit(url, "//")[[1]][2]
  mirror_path <- paste0(root_dir, rel_path)
  if(!dir.exists(mirror_path))
  {
    build_path <- root_dir
    for(i in strsplit(rel_path, "/")[[1]])
    {
      build_path <- paste0(build_path, i, "/")
      dir.create(build_path)
    }
    write.csv(links, paste0(mirror_path, ".mirrordat.csv"))
  }
  records <- read.csv(paste0(mirror_path, ".mirrordat.csv"), stringsAsFactors = FALSE)
  current_files <- records$x
  n_updated <- 0
  if(!silent) cat("Updating files - please wait")
  for(i in seq_along(links))
  {
    if(!(links[i] %in% current_files))
    {
      download.file(paste0(url, links[i]), paste0(mirror_path, links[i]))
      n_updated <- n_updated + 1
    }
  }
  if(!silent) message(paste("Downloaded", n_updated, "files"))
  write.csv(links, paste0(mirror_path, ".mirrordat.csv"))
}

To run the function in your case, you would just run:

local_mirror("https://ftp.ncbi.nlm.nih.gov/pub/pmc/")

and to run it as a constant "event monitor" in the background, you would place it inside a looping function like this:

listen_for_changes <- function(url, poll_every = 5, silent = TRUE)
{
  repeat
  {
    local_mirror(url, silent = silent)
    Sys.sleep(poll_every)
  }
}

Which you would just run with:

listen_for_changes("https://ftp.ncbi.nlm.nih.gov/pub/pmc/")

I faced a similar problem a while ago, and I wrote a (somewhat) comprehensive function called repchkdl() for this purpose, which I've now bundled into a small R package.

Here is a link to its Github repo.

It is similar to the solution suggested by @Allan Cameron in many ways, and provides additional options for fine tuning how the remote URL is scanned (manual/auto options), how files are downloaded (e.g., filtering with regular expressions), and iteration frequency + limits (e.g., scan every 2 minutes, repeat that twice in total and exit thereafter) among other things.

Example usage:

test <- repchkdl(inpurl = "ftp://speedtest.tele2.net/upload/", 
                 inpregex = ".txt$", autoscan = "y", autodl = "y", inpwait = 10,
                 scanlim = 2)

This will automatically download (autoscan = "y", autodl = "y") all files ending with .txt (inpregex = ".txt$") from the URL ftp://speedtest.tele2.net/upload/, and check for updates every 10 seconds (inpwait = 10). The function will re-check twice (scanlim = 2) before exiting. A vector of the filenames will be saved to the object test.

Edit: The package has been completely overhauled, and has additional features including silent mode operation. The repo location remains the same. The function invocation has changed slightly as follows (as an example):

test <- repchkdl(url = "ftp://ftp.swfwmd.state.fl.us/pub/usf/", 
                 usr = "anonymous", pwd = "exampleuser@example.com",
                 scanlim = 2, wait = 2, verbosity = TRUE)

head(test)
#                      flinks   fsizes dlstats
# 1        SWFWMD_May2020.txt 25321197 SUCCESS
# 2 USF_SCADA_AM_20200517.csv  2322041 SUCCESS
# 3 USF_SCADA_AM_20200518.csv  2321932 SUCCESS
# 4 USF_SCADA_AM_20200519.csv  2323118 SUCCESS
# 5 USF_SCADA_AM_20200520.csv  2322689 SUCCESS
# 6 USF_SCADA_AM_20200521.csv  2322243 SUCCESS