How can I "convert" ISO-8859-7 strings to UTF-8 in C++?

Viewed 827

I'm working with 10+ years old machines which use ISO 8859-7 to represent Greek characters using a single byte each. I need to catch those characters and convert them to UTF-8 in order to inject them in a JSON to be sent via HTTPS. Also, I'm using GCC v4.4.7 and I don't feel like upgrading so I can't use codeconv or such.

Example: "OΛΑ": I get char values [ 0xcf, 0xcb, 0xc1, ], I need to write this string "\u039F\u039B\u0391".

PS: I'm not a charset expert so please avoid philosophical answers like "ISO 8859 is a subset of Unicode so you just need to implement the algorithm".

3 Answers

Given that there are so few values to map, a simple solution is to use a lookup table.

Pseudocode:

id_offset    = 0x80  // 0x00 .. 0x7F same in UTF-8
c1_offset    = 0x20  // 0x80 .. 0x9F control characters

table_offset = id_offset + c1_offset

table = [
    u8"\u00A0",  // 0xA0
    u8"‘",       // 0xA1
    u8"’",
    u8"£",
    u8"€",
    u8"₯",
    // ... Refer to ISO 8859-7 for full list of characters.
]

let S be the input string
let O be an empty output string
for each char C in S
    reinterpret C as unsigned char U
    if U less than id_offset       // same in both encodings
        append C to O
    else if U less than table_offset  // control code
        append char '\xC2' to O  // lead byte
        append char C to O
    else
        append string table[U - table_offset] to O

All that said, I recommend to save some time by using a library instead.

One way could be to use the Posix libiconv library. On Linux, the functions needed (iconv_open, iconv and iconv_close) are even included in libc so no extra linkage is needed there. On your old machines you may need to install libiconv but I doubt it.

Converting may be as simple as this:

#include <iconv.h>

#include <cerrno>
#include <cstring>
#include <iostream>
#include <iterator>
#include <stdexcept>
#include <string>

// A wrapper for the iconv functions
class Conv {
public:
    // Open a conversion descriptor for the two selected character sets
    Conv(const char* to, const char* from) : cd(iconv_open(to, from)) {
        if(cd == reinterpret_cast<iconv_t>(-1))
            throw std::runtime_error(std::strerror(errno));
    }

    Conv(const Conv&) = delete;

    ~Conv() { iconv_close(cd); }

    // the actual conversion function
    std::string convert(const std::string& in) {
        const char* inbuf = in.c_str();
        size_t inbytesleft = in.size();

        // make the "out" buffer big to fit whatever we throw at it and set pointers
        std::string out(inbytesleft * 6, '\0');
        char* outbuf = out.data();
        size_t outbytesleft = out.size();

        // the const_cast shouldn't be needed but my "iconv" function declares it
        // "char**" not "const char**"
        size_t non_rev_converted = iconv(cd, const_cast<char**>(&inbuf),
                                         &inbytesleft, &outbuf, &outbytesleft);

        if(non_rev_converted == static_cast<size_t>(-1)) {
            // here you can add misc handling like replacing erroneous chars
            // and continue converting etc.
            // I'll just throw...
            throw std::runtime_error(std::strerror(errno));
        }

        // shrink to keep only what we converted
        out.resize(outbuf - out.data());

        return out;
    }

private:
    iconv_t cd;
};

int main() {
    Conv cvt("UTF-8", "ISO-8859-7");

    // create a string from the ISO-8859-7 data
    unsigned char data[]{0xcf, 0xcb, 0xc1};
    std::string iso88597_str(std::begin(data), std::end(data));

    auto utf8 = cvt.convert(iso88597_str);
    std::cout << utf8 << '\n';
}

Output (in UTF-8):

ΟΛΑ

Using this you can create a mapping table, from ISO-8859-7 to UTF-8, that you include in your project instead of iconv:

Demo

Ok I decided to do this myself instead of looking for a compatible library. Here's how I did.

The main problem was figuring out how to fill the two bytes for Unicode using the single one for ISO, so I used the debugger to read the value for the same character, first written by the old machine and then written with a constant string (UTF-8 by default). I started with "O" and "Π" and saw that in UTF-8 the first byte was always 0xCE while the second one was filled with the ISO value plus an offset (-0x30). I built the following code to implement this and used a test string filled with all greek letters, both upper and lower case. Then I realised that starting from "π" (0xF0 in ISO) both the first byte and the offset for the second one changed, so I added a test to figure out which of the two rules to apply. The following method returns a bool to let the caller know whether the original string contained ISO characters (useful for other purposes) and overwrites the original string, passed as reference, with the new one. I worked with char arrays instead of strings for coherence with the rest of the project which is basically a C project written in C++.

bool iso_to_utf8(char* in){
bool wasISO=false;

if(in == NULL)
    return wasISO;

// count chars
int i=strlen(in);
if(!i)
    return wasISO;

// create and size new buffer
char *out = new char[2*i];
// fill with 0's, useful for watching the string as it gets built
memset(out, 0, 2*i);

// ready to start from head of old buffer
i=0;
// index for new buffer
int j=0;
// for each char in old buffer
while(in[i]!='\0'){
    if(in[i] >= 0){
        // it's already utf8-compliant, take it as it is
        out[j++] = in[i];
    }else{
        // it's ISO
        wasISO=true;
        // get plain value
        int val = in[i] & 0xFF;
        // first byte to CF or CE
        out[j++]= val > 0xEF ? 0xCF : 0xCE;
        // second char to plain value normalized
        out[j++] = val - (val > 0xEF ? 0x70 : 0x30);
    }
    i++;
}
// add string terminator
out[j]='\0';
// paste into old char array
strcpy(in, out);

return wasISO;

}

Related