The documentation for tm explains this if you dig down (see ??tm::DublicCore). From the docs:
A corpus has two types of metadata. Corpus metadata ("corpus") contains corpus specific metadata in form of tag-value pairs. Document level metadata ("indexed") contains document specific metadata but is stored in the corpus as a data frame. Document level metadata is typically used for semantic reasons (e.g., classifications of documents form an own entity due to some high-level information like the range of possible values) or for performance reasons (single access instead of extracting metadata of each document). The latter can be seen as a from of indexing, hence the name "indexed". Document metadata ("local") are tag-value pairs directly stored locally at the individual documents.
DataframeSource automatically assigns only the the corpus metadata*. For example, see what the following prints:
library(tm)
data <- data.frame(doc_id = c(234345345, 1299),
text = c("The Prince and the Pauper",
"Little Women"),
author = c('Mark Twain', 'Louisa May Alcott'),
date = c(1881, 1868),
stringsAsFactors = FALSE)
samplecorpus <- Corpus(DataframeSource(data))
meta(samplecorpus)
# Or even
meta(samplecorpus[1], tag = 'author')
In order to assign metadata at the document level, you can work with meta to change tags. Bizarrely, this only works if you use VCorpus. So changing the above slightly, you can do:
samplecorpus <- VCorpus(DataframeSource(data))
# Can now set document metadata tags
meta(samplecorpus[[1]], tag = 'author') <- 'Mark Twain'
*EDIT:
Contemplating further (and responding to OP's comment), I agree that the documentation is not a completely accurate description of the package's observed behavior. The quoted documentation above refers to three levels (Corpus, indexed document level, and local document level), which in my example appear to correspond to samplecorpus, samplecorpus[1], and samplecorpus[[1]], respectively. If this correct, then the metadata is being assigned by DataframeSource at the promised level (if somewhat vaguely, as they never specified which document-level). However, the docs also claims the indexed document level is stored as a data frame and local as tag-value pairs, but both are stored as lists. Confusing. I can only conclude that this is either a bug in the package implementation or an error in the docs.
Barring contacting the package authors to clear this up (not a bad idea), I would propose the following workaround:
samplecorpus <- VCorpus(DataframeSource(data))
transfer_metadata <- function(x, i, tag){
return(meta(x[i], tag=tag)[[tag]])
}
tags <- colnames(data)
tags <- tags[! tags %in% c('doc_id', 'text')]
for(i in 1:length(samplecorpus)){
for (tag in tags){
meta(samplecorpus[[i]], tag=tag) <- transfer_metadata(samplecorpus, i=i, tag=tag)
}
}
