R not detecting \x pattern in string

Question

does someone know how it is possible de detect and replace "\x" in R?

library(stringr)
x <- "gesh\xfc"
str_detect(x, "\x")
# Error in stri_detect_regex(string, pattern, negate = negate, opts_regex = opts(pattern)) : 
# Unrecognized backslash escape sequence in pattern. (U_REGEX_BAD_ESCAPE_SEQUENCE)

nchar(x)
# Error in nchar(x) : invalid multibyte string, element 1

iconv(x, "latin1", "utf-8")
# [1] "geshü"
Encoding(x)
# [1] "unknown"

Session Info:

> sessionInfo()
R version 3.6.0 (2019-04-26)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS High Sierra 10.13.6

...

locale:
[1] fr_CH.UTF-8/fr_CH.UTF-8/fr_CH.UTF-8/C/fr_CH.UTF-8/fr_CH.UTF-8

Context: I read a .csv file with data.table::fread() but this file has colnames in German with letters such as ä,ö,ü,etc. Once read in R, those letters transform into something starting with "\x". This is simple unusable in R afterwards.

user_2372064 · Answer

Just to summarize what happened here. The "\x" is NOT part of the string. This is just how R escapes values that it can't otherwise print. In the case of "gesh\xfc", the first 4 characters are basic ASCII characters, but the last character is encoded is "\xfc". In the latin1 encoding (which Windows uses by default) the fc character is the "ü" character. So in my windows machine, I see

x <- "gesh\xfc"
x
# [1] "geshü"

And you can look at the raw bytes of that string with

charToRaw("gesh\xfc")
# [1] 67 65 73 68 fc

You can see the ASCII hex character codes for the first 4 values, and then you can see that the \x was actually just used to include the "fc" character code in the string. The string itself only has 5 "characters".

But if you are not using latin1, the "fc" character doesn't map to anything. Basically that string doesn't make any sense in the utf-8 encoding which is what the mac uses by default. You can convert to utf-8 with

iconv("gesh\xfc", "latin1", "utf-8")

But since you got this file by importing a text file, the problem was that the R didn't know the encoding of the file wasn't UTF-8 so you wound up with these weird values. You should tell fread that the file came from windows so it can import the strings properly from the start

fread(file, encoding = "Latin-1")

You need to know what encoding was used to make a file you are importing especially when made by someone else. It's not really possible for programs to guess correctly.

R not detecting \x pattern in string

1 Answers