Rvest/Stringr: Remove invisible symbols splitting words?

user_2221566 0
Viewed 126

I am detecting words with stringr and regex and noticed that I am missing several items because the words are split when there’s a linebreak.

Strikingly, the symbol breaking the word is not visible when I print the cell’s content or look at it in RStudio’s data viewer. I can only see that words were split when copying-pasting the content of e.g. the dataframe’s cell into the console. There is a little dot separating the word, e.g. Schallenberg is represented as Schallen[dot]berg (a little dot, vertically in the middle of the line);

enter image description here

I assume that the invisibility of the symbols splitting the words has something to do with the text’s origin. They were retrieved via rvest. However, even more strikingly when I check the words which contain the symbol on the website, they are not split.

My question: How can I remove these otherwise invisible symbols? They are not removed with stringr::str_squish. Do I have to account for them when already extracting the text from the website? Many thanks!

library(rvest, quietly = T, warn.conflicts = F)
library(tidyverse, quietly = T, warn.conflicts = F)

web_link <- "https://www.parlament.gv.at//PAKT/VHG/XXVII/NRSITZ/NRSITZ_00006/fnameorig_797359.html"

df_txt <- web_link %>% 
  read_html(., encoding = "latin1") %>% 
  html_nodes("body > div.WordSection37 > p:nth-child(2) > b:nth-child(2) > span") %>%
  html_text2() %>% 
  enframe(name = NULL,
          value="text_raw") %>% 
  mutate(text_raw=text_raw %>% str_squish %>% str_trim(., "both")) 

# There is a dot between Schallen - berg, but it's not visible
print(df_txt$text_raw)
#> [1] "Bundesminister für Europa, Integration und Äußeres Mag. Alexander Schallen­berg, LL.M."

str_detect(df_txt$text_raw, "Schallenberg") #false
#> [1] FALSE

Created on 2021-04-08 by the reprex package (v1.0.0)

stringr rvest r regex
1 Answers

This character you got in the input is a soft hyphen, \xAD. It belongs to a Other, format Unicode control char category. So, you can use

gsub("\\p{Cf}+", "", df_txt$text_raw, perl=TRUE)

Or, with stringr:

stringr::str_remove_all(df_txt$text_raw, "\\p{Cf}+")
stringr::str_replace_all(df_txt$text_raw, "\\p{Cf}+", "")

NOTE: This won't remove all invisible chars. You might want to remove any control chars other than common line break chars, \r and \n. In this case, you'd rather use

gsub("[^\\P{C}\r\n]+", "", "a\n\r\\b", perl=TRUE)
stringr::str_remove_all(df_txt$text_raw, "[^\\P{C}\r\n]+")
stringr::str_replace_all(df_txt$text_raw, "[^\\P{C}\r\n]+", "")

where [^\\P{C}\r\n]+ is a negated character class that matches any char but a non-control char, CR and LF chars (that is, it matches any control char but \r and \n chars).

user_38329700
Related