identical strings from different data files won't match in R

user_190352 0
Viewed 977

I was doing some file merging, and two files wouldn't - despite having a key column that matches (I actually generated one key column by copy-pasting from the other). It's the damndest thing, and I worry that I'm either going crazy or missing something fundamental. As an example (and I cannot figure out how to make it reproducible, as when I copy and paste these strings into new objects, they compare just fine), here's my current console:

> q
[1] "1931 80th Anniversary"
> z
[1] "1931 80th Anniversary"
> q == z
[1] FALSE

I str-ed both, just in case I missed something, and...

> str(q)
 chr "1931 80th Anniversary"
> str(z)
 chr "1931 80th Anniversary"

What could be going on here?

string-matching r
2 Answers

This was a great puzzler. To answer - to diagnose the problem, charToRaw() was the answer.

> charToRaw(q)
 [1] 31 39 33 31 c2 a0 38 30 74 68 c2 a0 41 6e 6e 69 76 65
[19] 72 73 61 72 79
> charToRaw(z)
 [1] 31 39 33 31 20 38 30 74 68 20 41 6e 6e 69 76 65 72 73
[19] 61 72 79

Oh! Different! It seems to lie in the encoding, which, given that these were both plain ole' CSVs I loaded from, I never would have guessed, but

> Encoding(q)
[1] "UTF-8"
> Encoding(z)
[1] "unknown"

In the end, I used iconv() on q to make it work

> iconv(q, from = 'UTF-8', to = 'ASCII//TRANSLIT') == z
[1] TRUE

This has been a weird journey, and I hope this helps someone else who is as baffled as I was - and they learn a few new functions along the way.

user_1903520

It looks like you have non-breaking spaces in your string, which isn't really an encoding issue. This happens to me all the time because alt + space inserts a non-breaking space on a Mac, and I use alt on my German keyboard for all sorts of special characters, too. My pinkies are my slowest fingers and they don't always release alt fast enough when I transition from some special character to a space. I discovered this problem writing bash scripts, where <command> | <command> is common and | is alt + 7.

I think stringr::str_replace_all(q, "\\s", " ") should fix your current issue. Alternatively, you can try targeting specific non-printables, e.g. in your situation stringr::str_replace_all(q, "\uA0", " "). To expose the offending characters you can use stringi::stri_escape_unicode(q), which would return "1931\\u00a080th\\u00a0Anniversary". You can then just copy and paste to get the same results as above: stringr::str_replace_all(q, "\u00a0", " ")

Admin1
Related