Problem downloading a dataset from a wlatin1 environment to a UTF-8 one

CarLaTeX 0
Viewed 38

I'm downloading (with a proc download in SAS) a dataset from a server with wlatin1 encoding to a server with UTF-8 encoding.

I've got the error

ERROR: Some character data was lost during transcoding in the data set
       libref.datasetname. Either the data contains characters that are not
       representable in the new encoding or truncation occurred during transcoding.

I tried setting inencoding='utf8' or inencoding='asciiany' on the input dataset but it doesn't work (maybe because the wlatin1 server has SAS 9.3 whereas the UTF-8 server has SAS 9.4).

Rewriting the file like in the following code (and then doing the proc download of myoutput) works, but I was wondering if there is a more elegant way to do the same thing.

data myoutput;
set pathin.myinput;
/*Translittera la I accentata con I normale*/
des_nome = tranwrd(des_nome,'CD'x,'I');
des_nome = tranwrd(des_nome,'ED'x,'i');
/*Translittera la A accentata con A normale*/
des_nome = tranwrd(des_nome,'C1'x,'A');
des_nome = tranwrd(des_nome,'E1'x,'a');
/*Translittera la E accentata con E normale*/
des_nome = tranwrd(des_nome,'C9'x,'E');
des_nome = tranwrd(des_nome,'E9'x,'e');
/*Translittera la O accentata con O normale*/
des_nome = tranwrd(des_nome,'D2'x,'O');
des_nome = tranwrd(des_nome,'D3'x,'O');
des_nome = tranwrd(des_nome,'D6'x,'O');
des_nome = tranwrd(des_nome,'F3'x,'o');
/*Translittera la U accentata con U normale*/
des_nome = tranwrd(des_nome,'DC'x,'U'); 
des_nome = tranwrd(des_nome,'F9'x,'u'); 
/*Translittera la Y accentata con Y normale*/
des_nome = tranwrd(des_nome,'DD'x,'Y');
des_nome = tranwrd(des_nome,'FD'x,'y');
/*Translittera accenti strani con '*/
des_nome = tranwrd(des_nome,'B4'x,"'");
/*Translittera simboli strani con spazi*/
des_nome = tranwrd(des_nome,'A7'x,' ');  /* § nel NOME */
des_nome = tranwrd(des_nome,'A3'x,' ');  /* £ nel NOME */
cod_cap_res = tranwrd(cod_cap_res,'A3'x,' '); /* £ nel CAP */
run;
sas character-encoding encoding
3 Answers
Stu Sztukowski0

This issue is that the storage length of at least one of the variables in original dataset is too short to fit the expanded length required by the UTF-8 representation of at least one of the strings.

Here is simple way to demonstrate the problem. Create a simple file with all 256 possible characters of the WLATIN1 (or LATIN1) encoding.

340  %put %sysfunc(getoption(encoding,keyword));
ENCODING=WLATIN1
341  data 'c:\downloads\wlatin1.sas7bdat';
342    string = collate(0,256);
343  run;

NOTE: The data set c:\downloads\wlatin1.sas7bdat has 1 observations and 1 variables.
NOTE: DATA statement used (Total process time):
      real time           0.00 seconds
      cpu time            0.00 seconds

Now try to read it in a session using UTF-8 encoding. Even if you make the variable longer in the output dataset the conversion fails.

38   data test1;
39     length string $1024;
40     set 'c:\downloads\wlatin1.sas7bdat';
NOTE: Data file WC000001.WLATIN1.DATA is in a format that is native to another host, or the file encoding does not match the
      session encoding. Cross Environment Data Access will be used, which might require additional CPU resources and might reduce
      performance.
41   run;

ERROR: Some character data was lost during transcoding in the dataset WC000001.WLATIN1. Either the data contains characters that
       are not representable in the new encoding or truncation occurred during transcoding.
NOTE: The DATA step has been abnormally terminated.
NOTE: The SAS System stopped processing this step because of errors.
WARNING: The data set WORK.TEST1 may be incomplete.  When this step was stopped there were 0 observations and 1 variables.
WARNING: Data set WORK.TEST1 was not replaced because this step was stopped.
NOTE: DATA statement used (Total process time):
      real time           0.01 seconds
      cpu time            0.00 seconds

So to fix that read the file using ENCODING='ANY' and then convert the strings from WLATIN1 to UTF-8.

42   data test;
43     length string $1024;
44     set 'c:\downloads\wlatin1.sas7bdat' (encoding='any');
45     string = kcvt(string,'wlatin1','utf-8');
46   run;

NOTE: There were 1 observations read from the data set c:\downloads\wlatin1.sas7bdat.
NOTE: The data set WORK.TEST has 1 observations and 1 variables.
NOTE: DATA statement used (Total process time):
      real time           0.01 seconds
      cpu time            0.01 seconds
user7NPk0

You might have better luck letting the newer version of SAS that is running with UTF-8 encoding do the transcoding instead of forcing the remote libref engine to attempt to deal with it.

So if you wanted to download the dataset MYLIB.MYDATA you could first copy the actual dataset file. Then transcode it.

%syslput lwork=%qsysfunc(pathname(work));
rsubmit;
  proc download binary infile="%sysfunc(pathname(MYLIB))/mydata.sas7bdat"
     outfile="&lwork/mydata.sas7bdat" ;
  run;
endrsubmit;
data mydata; 
  set mydata(encoding='wlatin1');
run;
user7NPk0
Related