NSString initWithData returns null

user_68589 0
Viewed 23860

I am pulling data from a website via NSURLConnection and stashing the received data away in an instance of NSMutableData. In the connectionDidFinishLoading delegate method the data is convert into a string with a call to NSString's appropriate method:

NSString *result = [[NSString alloc] initWithData:data 
                                     encoding:NSUTF8StringEncoding]

The resulting string turns out to be a null. If I use the NSASCIIStringEncoding, however, I do obtain the appropriate string, albeit with unicode characters garbled up as expected. The server's Content-Type header does not specify the UTF-8 encoding, but I have attempted a number of different websites with a similar scenario, and there string conversion happens just fine. It seems like the problem only pertains to the given web service but I have no clue why.

On a side note, is pulling web pages and data from an API good practice, i.e. buffering the data, converting into a string, and manipulating the string afterwards?

Much appreciated!

nsstring macos objective-c cocoa iphone
5 Answers

As Peter said, the content-type Header is just an "hint" of what the content sent is expected to be. On server side you can set any content-type and send any bytes sequences, which can be invalid.

I had exactly the same issue dealing with incorrect UTF-8 data, which included ISO-8859-1 (Latin-1) characters (french accents).

Wikipedia about UTF-8 is worth reading to understand this issue and how to handle encoding errors.

The fact is that NSString initWithData:encoding: strict implementation just return nil when a decoding error occurs. (unlike java for instance which use a replacement character)

The peter solution of converting a mostly UTF-8 data into Latin-1 was not satisfying me. (All UTF-8 characters becomes incorrect, for just one Latin 1 erratic character)

Best option would be a fix on server side, sure, but I'm not responsible on this side...

So I looked deeper, and found a solution using GNU libiconv C library (available on OSX and iOS) The principle is using iconv to remove non UTF-8 invalid characters (i.e. "prété" will become "prt")

Here is a sample code, equivalent of the command line iconv -c -f UTF-8 -t UTF-8 invalid.txt > cleaned.txt

#include "iconv.h"

- (NSData *)cleanUTF8:(NSData *)data {
  iconv_t cd = iconv_open("UTF-8", "UTF-8"); // convert to UTF-8 from UTF-8
  int one = 1;
  iconvctl(cd, ICONV_SET_DISCARD_ILSEQ, &one); // discard invalid characters

  size_t inbytesleft, outbytesleft;
  inbytesleft = outbytesleft = data.length;
  char *inbuf  = (char *)data.bytes;
  char *outbuf = malloc(sizeof(char) * data.length);
  char *outptr = outbuf;
  if (iconv(cd, &inbuf, &inbytesleft, &outptr, &outbytesleft)
      == (size_t)-1) {
    NSLog(@"this should not happen, seriously");
    return nil;
  }
  NSData *result = [NSData dataWithBytes:outbuf length:data.length - outbytesleft];
  iconv_close(cd);
  free(outbuf);
  return result;
}

Then the resulting NSData can be safely decoded using NSUTF8StringEncoding

Note that latest iconv also allow fallback methods by using :

iconvctl(cd, ICONV_SET_FALLBACKS, &fallbacks);

By using a fallback on unicode errors, you can use a replacement character, or better, to try another encoding. In my case I managed to fallback to LATIN-1 where UTF-8 failed, which resulted in 99% positive conversions. Look at iconv source code for understanding it.

user_5178320
Related