Looking for a more effective approach to scan and remove text in a large data frame

user_19516924 0
Viewed 81

I'm working on a sentiment analysis of tweets, but unfortunately one of the pre-processing steps in my code are not very effective.

The code is a for loop that looks for all words starting with a hashtag and then search for the word in a large vector of about 52,000 words from the Reuters dataset. If it finds the word in the vector, it removes the hashtag, but keeps the word.

If the for loop does not find the word in the vector, the next code removes both the hashtag and the word.

The tweets are in data_tweet, which is a data frame with a few million rows.

library(qdap)
library(tm.corpus.Reuters21578)

data("Reuters21578")

# Makes a list of all words in Reuters and how often they are used
freq_terms <- freq_terms(Reuters21578, Inf)

# Makes a vector with all unique words from Reuters
all_words_vec <-  as.vector(freq_terms$WORD)

# The for loop which is by far the most time consuming part of the code
# Searching for words starting with # and delete them if found in Reuters vector
for (i in 1:length(all_words_vec)) {  
  data_tweet$text <- gsub(paste0("#\\<", all_words_vec[[i]], "\\>"), paste(all_words_vec[[i]], ""), data_tweet$text)  
  (i %% 10000 == 0)  
  cat(i)  
  cat("..")  
}

# Remove tokens that start with #
data_tweet$text <- gsub("#\\S+", "", data_tweet$text)

Is there a more effective approach?

tweets sentiment-analysis r
1 Answers

For loops are generally speaking not used in R, and are quite slow with your current implementation. You can try using purrr::walk. Walk is used instead of map when you are not trying to return a unit of information, but are doing some task like reassigning a variable or writing a csv.

purrr::walk(all_words_vec, function(x) {data_tweet$text <- gsub(paste0("#\\<", x, "\\>"), paste(x, ""), data_tweet$text)})
    
data_tweet$text <- gsub("#\\S+", "", data_tweet$text)
user_195126110
Related