How to import multiple Factiva html files by loop

Question

I am using the Factiva Package ‘tm.plugin.factiva’ to import html files containing a Factiva search. It has worked beautifully so far, but now I have a problem with importing data and constructing a corpus from several html files (350 in total). I cannot figure out how to write a loop to iterate the simple step-by-step import code I have used before.

Earlier, with a smaller sample, I have managed to import the html files an a step-by-step process:

library(R.temis)
library(tm)
library(tm.plugin.factiva)

# Import corpus
source1 <- FactivaSource("Factiva1.html")
source2 <- FactivaSource("Factiva2.html")
source3 <- FactivaSource("Factiva3.html")

corp_source1 <- Corpus(source1, list(language=NA))
corp_source2 <- Corpus(source2, list(language=NA))
corp_source3 <- Corpus(source3, list(language=NA))

full_corpus <- c(corp_source1, corp_source2, corp_source3)

However, this is obviously not an option for the 350 html files. I have tried writing a loop for the import:

# Import corpus
files <- list.files(my_path)

for (i in files){
  source <- FactivaSource(i)
}

tech_corpus <- Corpus(source, list(language=NA))

And:

htmlFiles <- Sys.glob("Factiva*.html")

for (k in 1:lengths(htmlFiles[[k]])){
  source <- FactivaSource(htmlFiles[[k]])
}

But both of these only reads the first html file into the source, not the rest.

I have also tried:

for (k in seq_along(htmlFiles)){
  source <- FactivaSource(htmlFiles[1:k], encoding = "UTF-8", format = c("HTML"))
}

But then I get the error that:

Error: x must be a string of length 1. I have tried manipulating the htmlFiles into a list (by: html_list <- as.list(htmlFiles)), but no change in result.

The two loops that did work, but only for the first html file.

I got the same result when I tried looping constructing the corpus as well.

for (m in 1:lengths(htmlFiles)){
  corp_source <- Corpus(htmlFiles[[m]], list(language=NA))
}

Which worked, but only for the first html file. But then I get the error:

In 1:lengths(htmlFiles) : numerical expression has 5 elements: only the first used

I would highly appreciate any help to understand how to get around this issue. Ideally, a loop to repeat the step-by-step process I did in the beginning would be super, as it seems to me that neither the FactivaSource() or Corpus() likes the complications I have made here - but I am far from an expert. Any help will be highly appreciated!

How to import multiple Factiva html files by loop

0 Answers