R process taking increasingly more memory on linux

Viewed 41

I am running an R script on an AWS EC2 instance. The script reads in a list of species names, and then determines how long the wikipedia page is for each species (using rvest) and also checks how many page visits that page has had (using the wikipedia api via the pageviews package), before writing that to disk. Here is the script:

f <- read.csv('species.csv', header=F, stringsAsFactors=F)

library(rvest)
library(dplyr)
library(RCurl)
library(pageviews)

for (i in 1:length(f$V1)){
  tryCatch({
    t <- gsub(' ', '_', f$V1[i])

    url <- paste0("http://en.wikipedia.org/wiki/", t)

    if (url.exists(url)){
      out <- read_html(url) %>%
        html_text %>%
        nchar

      views <- article_pageviews(project='en.wikipedia', article = t)

      write.csv(data.frame(species=f$V1[i], size=out, views=views$views), paste0('AllSpecies/', t, '.csv'), row.names=F)
    }

    cat(paste0(round(i/nrow(f)*100, 4), '%'), Sys.time(), f$V1[i], '\n')

  }, error=function(e){cat(f$V1[i], '\n', file='errors.txt', append=T)})
}

After noticing I was having memory issues, I modified the code to write after each species, rather than accumulate all of the results in a large dataframe and write it all at the end, but R is still having issues.

I ran a shell script to output the results of ps every ten minutes, and it is definitely this one R process that is taking all of the memory. %MEM starts at about 12% and then increases by about 6% an hour until the system kills the R process.

This is a kind of error that I have very little experience debugging. I'm not sure if it is something in my code, or if it might be something from one of the packages I am using.

Edit: f is just a dataframe with one character column. It's has 0.5 million rows, but here is the output of dput(f[1:100, , drop=F]):

structure(list(V1 = c("Eschweilera coriacea", "Eschweilera costaricensis",
"Eschweilera cyathiformis", "Eschweilera decolorans", "Eschweilera eperuetorum",
"Eschweilera fanshawei", "Eschweilera gigantea", "Eschweilera grandiflora",
"Eschweilera hondurensis", "Eschweilera integricalyx", "Eschweilera integrifolia",
"Eschweilera itayensis", "Eschweilera jacquelyniae", "Eschweilera juruensis",
"Eschweilera klugii", "Eschweilera laevicarpa", "Eschweilera longipedicellata",
"Eschweilera longirachis", "Eschweilera macrocarpa", "Eschweilera mattos-silvae",
"Eschweilera mexicana", "Eschweilera micrantha", "Eschweilera microcalyx",
"Eschweilera nana", "Eschweilera neblinensis", "Eschweilera neei",
"Eschweilera obversa", "Eschweilera ovalifolia", "Eschweilera ovata",
"Eschweilera pachyderma", "Eschweilera panamensis", "Eschweilera paniculata",
"Eschweilera parviflora", "Eschweilera parvifolia", "Eschweilera pedicellata",
"Eschweilera perumbonata", "Eschweilera piresii", "Eschweilera pittieri",
"Eschweilera potaroensis", "Eschweilera praealta", "Eschweilera pseudodecolorans",
"Eschweilera punctata", "Eschweilera rabeliana", "Eschweilera rankiniae",
"Eschweilera revoluta", "Eschweilera rhododendrifolia", "Eschweilera rhodogonoclada",
"Eschweilera rimbachii", "Eschweilera rionegrense", "Eschweilera rodriguesiana",
"Eschweilera roraimensis", "Eschweilera rufifolia", "Eschweilera sagotiana",
"Eschweilera sclerophylla", "Eschweilera sessilis", "Eschweilera simiorum",
"Eschweilera squamata", "Eschweilera subcordata", "Eschweilera subglandulosa",
"Eschweilera tenax", "Eschweilera tenuifolia", "Eschweilera tessmannii",
"Eschweilera tetrapetala", "Eschweilera truncata", "Eschweilera venezuelica",
"Eschweilera wachenheimii", "Foetidia africana", "Foetidia asymetrica",
"Foetidia capuronii", "Foetidia cuneata", "Foetidia delphinensis",
"Foetidia dracaenoides", "Foetidia macrocarpa", "Foetidia parviflora",
"Foetidia pterocarpa", "Foetidia rodriguesiana", "Foetidia rubescens",
"Foetidia sambiranensis", "Foetidia vohemarensis", "Grias cauliflora",
"Grias colombiana", "Grias haughtii", "Grias longirachis", "Grias multinervia",
"Grias neuberthii", "Grias peruviana", "Gustavia acuminata",
"Gustavia acuta", "Gustavia angustifolia", "Gustavia augusta",
"Gustavia brachycarpa", "Gustavia coriacea", "Gustavia dodsonii",
"Gustavia dubia", "Gustavia elliptica", "Gustavia erythrocarpa",
"Gustavia excelsa", "Gustavia flagellata", "Gustavia foliosa",
"Gustavia fosteri")), .Names = "V1", row.names = c(NA, 100L), class = "data.frame")
0 Answers
Related