how to convert utf-8 to utf-16 with ndash?

Question

both:

$result = iconv('UTF-8', 'UTF-16LE//IGNORE//TRANSLIT', $str);

and mb_convert_encoding() fail to convert a ndash (–) the long minus.

The result will go into a csv, so replacing it with an html entity is not an option. Any ideas?

code:

            $data = $eventHelper->getProgramForCsvExport($event);

            $response = new StreamedResponse();
            $response->setCharset('UTF-16LE');
            $filename = 'program-' . $event->getShortName() . $event->getShortYear() . '.csv';

            $utf16Data = [];
            foreach ($data as $row) {
                $utf16row = [];
               foreach ($row as $entry) {
                   $utf16row[] = iconv('UTF-8', 'UTF-16LE//IGNORE//TRANSLIT', $entry);
               }
                $utf16Data[] = $utf16row;

            }

            $response->setCallback(function () use ($utf16Data) {
                $output = fopen('php://output', 'w+');

                foreach ($utf16Data as $row) {
                    fputcsv($output, $row, ';');
                }

                fclose($output);
            });

            $response->headers->set('Content-Type', 'text/csv; charset=utf-16');
            $response->headers->set('Content-Disposition', 'attachment; filename="' . $filename . '"');

            return $response;

Edit: It's an export for the great office 365 which doesn't support UTF-8 anymore but is on UTF-16LE on default (as far as I read). German umlauts (äöüß) work fine (and didn't before the convert), but ndash (and maybe some other special chars) don't. Ndashes are either blank (on mac) or become brackets (or so) on windows.

user_157247 · Answer

I think the problem is that your code isn't outputting a UTF-16LE BOM (byte order mark) at the beginning of the file, so the programs reading it don't know what encoding it's in and are (apparently) guessing poorly.

A UTF-16LE BOM is the byte sequence 0xFF 0xFE (in that order) right at the beginning of the file. Make that the first thing you write to your output. More about BOMs in this Unicode FAQ.

To test my theory, I wrote the byte sequence for a UTF-16LE file containing only the characters 0–0:

FF FE 30 00 13 20 30 00

The FF FE is the BOM, the 30 00 is the digit zero, the 13 20 is the N-dash, and the final 30 00 is the final digit zero. (The zeros are just there so I can easily find the dash, though in such a short file it wouldn't really be difficult.)

I was able to open that with Office 365 on Windows just fine.

Then I wrote a file without the BOM:

30 00 13 20 30 00

Office 365 did indeed misinterpret the N-dash and show it as a character that looks like a pair of brackets.

how to convert utf-8 to utf-16 with ndash?

1 Answers