ruby & nlp: how to remove stop words and non-words (e.g. link, emoji) and count uniq words?

user_1794744 0
Viewed 191

Some sample data

NEW LISTING: @AaveAave AAVE ($LEND) will SOON be listed on @OKEx! Quiz & Net Buy up to 3,000 $LEND: ▶️ Follow us + @AaveAave ▶️ Join Quiz: https://t.co/{.....} ▶️ RT answer & @OKEx #OKExDeFi #OKExAave ▶️ Deposit + Net Buying rebate Listing details: https://t.co/{.....} hhttps://t.co/{.....}

100,000,000 #USDT (100,568,399 USD) transferred from Tether Treasury to #Binance Tx: https://t.co/{.....}

What I plan to do

  • remove all stop words
  • remove all non-words (link and emoji)
  • count word number and unique word number

My question: I know how to do this in R (using tidytext), but what is the best practice doing the above in Ruby? I searached around but didn't know any popular related gems.

Thanks to your help

If it's tidytext, the above can be pretty much done as shown below

enter image description here

nlp ruby
1 Answers

Take an Incremental Approach to Sanitizing Your Input

Your question is non-trivial because you haven't really defined "stop words" or "non-words" in a programmatic way, and there's no universal list of such. There isn't even a 100% accurate way to detect all valid web URIs, much less URIs using other schemes, although there are some "good enough and better than most" approaches.

However, as a basic starting point for rolling your own:

stop_words = /\b(a|an|the|and|or)\b/
non_words  = /\s+(#\p{ASCII}|https?:|@).*?\s+/

str = <<~'EOF'
  NEW LISTING: @AaveAave AAVE ($LEND) will SOON be listed on @OKEx! Quiz &amp; Net Buy  up to 3,000 $LEND: ▶️ Follow us + @AaveAave ▶️ Join Quiz: https://t.co/{.....} ▶️ RT answer &amp; @OKEx #OKExDeFi #OKExAave ▶️ Deposit + Net Buying rebate Listing details: https://t.co/{.....} hhttps://t.co/{.....}

            100,000,000 #USDT (100,568,399 USD) transferred from Tether Treasury to #Binance Tx: https://t.co/{.....} 
EOF

str.gsub! stop_words, ""
str.gsub! non_words, ""
str.gsub! /[[:^ascii:]]/, ""
str.strip!
str.squeeze!

p "Chars: #{str.length}"
#=> "Chars: 246"

p "Words: #{str.split.uniq.count}"
#=> "Words: 35"

There's still some further cleanup you'd need to do because your input is not standardized. Still, this should provide you a good starting point and a reasonable approach to finding your own "good enough" solution.

user_13019720
Related