Why do these identical Unicode strings have different serialized forms?

user_1451346 0
Viewed 63

When I compare a string literal with a non-ASCII character in R source code to the same string passed in through the command line, the two strings test as identical and have identical charToRaw representations, but serialize differently. What's going on? Aren't both strings in UTF-8?

To reproduce this, try this shell script (I'm running Dash 0.5.10.2-7 and R 4.0.2 on Ubuntu 20.10):

echo 'x="α"' > program.R
R --vanilla -q -e \
'source("program.R")
print(identical(x, "α"))
print(identical(charToRaw(x), charToRaw("α")))
print(identical(serialize(x, NULL), serialize("α", NULL)))'

which produces

> source("program.R")
> print(identical(x, "α"))
[1] TRUE
> print(identical(charToRaw(x), charToRaw("α")))
[1] TRUE
> print(identical(serialize(x, NULL), serialize("α", NULL)))
[1] FALSE
r character-encoding serialization
1 Answers

This might show what's going on

echo 'x="α"' > program.R
R --vanilla -q -e \
'source("program.R")
y="α"
.Internal(inspect(x))
.Internal(inspect(y))'

which for me gives

> .Internal(inspect(x))
@558f4edc1630 16 STRSXP g0c1 [REF(3)] (len=1, tl=0)
  @558f4edc15f8 09 CHARSXP g0c1 [REF(1),gp=0x20] [cached] "α"
> .Internal(inspect(y))
@558f4edd3860 16 STRSXP g0c1 [REF(2)] (len=1, tl=0)
  @558f4edd3898 09 CHARSXP g0c1 [REF(3),gp=0x28,ATT] [UTF8] [cached] "α"

So the values of x and y are different. I'm not an expert on all these values, but the gp= values are showing the string attributes. Those differences are what are resulting in the serialization differences. As far as I can tell, the string you set when you source the file has an undetermined encoding. It is not assumed to be UTF-8. If when you source() the file you specify the encoding explicitly, then the serializations will be the same.

So when I run

echo 'x="α"' > program.R
R --vanilla -q -e \
'source("program.R", encoding="UTF-8")
print(identical(x, "α"))
print(identical(charToRaw(x), charToRaw("α")))
print(identical(serialize(x, NULL), serialize("α", NULL)))'

I get

> print(identical(x, "α"))
[1] TRUE
> print(identical(charToRaw(x), charToRaw("α")))
[1] TRUE
> print(identical(serialize(x, NULL), serialize("α", NULL)))
[1] TRUE

Thus, replacing source("program.R") with source("program.R", encoding="UTF-8") makes the last check come out TRUE. Similarly, you can add x <- enc2utf8(x) or Encoding(x) <- "UTF-8".

user_23720640
Related