Converting packed ascii to ascii and back

Viewed 1121

I'm developing an application to communicate with a serialport device. The device itself is developed according to HART protocol.

Some documentation that I found related to this:

Packed ASCII is a subset of full ASCII and uses only 64 of the 256 possible characters. These 64 characters are the capitalized alphabet, numbers 0 through 9, and a few punctuation marks. Many HART parameters need only this limited ASCII set, which means that data can be compressed to 3/4 of normal.

The rule for converting from ASCII to Packed ASCII is just to remove bits 6 and 7 (two most significant).
The rules for conversion from packed ASCII back to ASCII are (1) set bit 7 = 0 and (2) set bit 6 = complement of packed ASCII bit 5.

So what I'm trying to do is get a value from the device, and later send it back, that is 6 bytes long and contains 8 characters and numbers. The value I get from unpacking is obviously incorrect. Don't know if the issue in this test is in the packing or unpacking. I'm not that familiar working with bytes so there might be some very obvious errors.

import java.nio.ByteBuffer;
import java.util.Arrays;

public class AsciiTest{

    public static void main(String []args){

        String text = "ABCDEF12";
        byte[] bytesFromText = text.getBytes();
        byte[] packed = packAscii(bytesFromText);
        byte[] unpacked = unpackAscii(packed);

    }


    //Function to pack bytes from 8 byte to 6 byte

    private static byte[] packAscii(byte[] bytes) {

        byte[] shorten = new byte[6];
        ByteBuffer packedAscii = ByteBuffer.wrap(shorten);
        int index = 0;

        for (int i = 0; i < bytes.length; i = i + 8) {
            int result = (int)(
                ((bytes[index] & 0x3F) << 42) +
                ((bytes[index + 1] & 0x3F) << 36) +
                ((bytes[index + 2] & 0x3F) << 30) +
                ((bytes[index + 3] & 0x3F) << 24) +
                ((bytes[index + 4] & 0x3F) << 18) +
                ((bytes[index + 5] & 0x3F) << 12) +
                ((bytes[index + 6] & 0x3F) << 6) +
                (bytes[index + 7] & 0x3F)
                );

            byte[] packedTemp = ByteBuffer.allocate(6).putInt(result).array();

            for (int j = 0; j < 6; j++)
            {
                packedAscii.put(packedTemp[j]);
            }

            index += 8;

        }

        byte[] combined = packedAscii.array(); 
        System.out.println(bytesToHex(combined)); // As hex string: "C41470920000"
        return combined;

    }

    // Function to pack bytes from 8 byte to 6 byte

    private static byte[] unpackAscii(byte[] bytes) {
        byte[] slice = bytes;
        byte[] nbytes = new byte[8];
        byte[] first = new byte[1];

        ByteBuffer buffer = ByteBuffer.wrap(nbytes);
        buffer.put(first);
        byte second = slice[0];
        second = (byte)~second;
        buffer.put(second);
        buffer.put(slice);

        byte[] combined = buffer.array();

        String hex = bytesToHex(combined2); 
        System.out.println(hex); // As hex string: "003BC41470920000"

        StringBuilder output = new StringBuilder();
        for (int i = 0; i < hex.length(); i+=2) {
            String str = hex.substring(i, i+2);
            output.append((char)Integer.parseInt(str, 16));
        } 
        
        System.out.println(output); // Output of string builder: ";Äp’"
        return combined; 
    }

    private final static char[] hexArray = "0123456789ABCDEF".toCharArray();
    public static String bytesToHex(byte[] bytes) {
        char[] hexChars = new char[bytes.length * 2];
        for ( int j = 0; j < bytes.length; j++ ) {
            int v = bytes[j] & 0xFF;
            hexChars[j * 2] = hexArray[v >>> 4];
            hexChars[j * 2 + 1] = hexArray[v & 0x0F];
        }
        return new String(hexChars);
    }
}

EDIT: Should have started with this, but here's the data from device. From device packed

01001000
01010000
11001001
01000000
01011000
00110001

By hand, unpacked the bits and then checked what the characters are.

543210   7654 3210

010010 | 0101 0010 R
000101 | 0100 0101 E
000011 | 0100 0011 C
001001 | 0100 1001 I
010000 | 0101 0000 P
000101 | 0100 0101 E
100000 | 0010 0000 Space
110001 | 0011 0001 1

Byte array from the packed bytes [72, 80, -55, 64, 88, 49]

So assuming that I got the binary representation correct (which the actual word suggests), then at least I have correct idea what to do. I'll update the code when I get progress there.

And the application is made for Android phones.

1 Answers

I was not checking actually issues in your code and decided to implement this from scratch. The solution is working but may be not too clean.

So, in packing mode, resulting first byte contains 6 bits of the 1st unpacked byte + 2 lower bits of the 2nd unpacked byte (thus shifted left by 6).
The 2nd resulting byte contains 4 higher bits of the 2nd unpacked byte (thus shifted right by 2) plus 4 lower bits of the 3rd unpacked byte (thus shifted left by 4)
Finally, the 3rd resulting byte contains 2 higher bits of the 3rd unpacked byte (shifted right by 6) plus the 4th unpacked byte shifted left by 2.

In the unpacking mode the process is reverse.

For packing mode 32 may be subtracted from each byte and in unpacking this constant is added back so that A, B..Z could fit into 6 bits properly.

update
Subtraction of 32 may be omitted, and there's an alternate way to restore "lost" bit 6 when unpacking for the values below 0x20 by adding 0x40.

Thus, the packing method is as follows:

private static byte[] packAscii(byte[] unpacked) {
    for (int i = 0; i < unpacked.length; i++) {
        // unpacked[i] = (byte)((unpacked[i] - 32) &0x3F);
        unpacked[i] = (byte)(unpacked[i] & 0x3F); // keep 6 lower bits as is
    }
    int size = (int) Math.ceil(unpacked.length * 3.0 / 4.0);
    byte[] result = new byte[size];
    // i - index of input unpacked array, r - index of result packed array 
    for (int i = 0, r = 0; i < unpacked.length;) {
        for (int j = 0; j < 4 && i < unpacked.length; j++, i++, r++) {
            int rightShift = j * 2;
            int leftShift = (8 - rightShift) % 8;
            switch(j) {
                case 0:
                    result[r]  = unpacked[i];
                    break;
                case 1:
                case 2:
                    result[r-1] |= (byte)(unpacked[i] << leftShift);
                    result[r]  = (byte)(unpacked[i] >> rightShift);
                    break;
                case 3:
                    result[r-1] |= (byte)(unpacked[i] << leftShift);
                    break;
            }
        }
        r--; // correction for the last packed byte
    }
    System.out.println("packed =" + printArr(result));
    return result;
}

Unpacking method is as follows:

final static int[][] masks = {
    {0x3F, 0x3F},
    {0x03, 0x3C},
    {0x0F, 0x30},
    {0x3F, 0x3F}
};

private static byte[] unpackAscii(byte[] packed) {
    int size = (int) Math.floor(packed.length * 4.0 / 3.0);
    byte[] result = new byte[size];
    System.out.println("unpacked size=" + size);
    for (int i = 0, r = 0; i < packed.length;) { // r index of unpacked array, i - packed
        for (int j = 0, rightShift = 6; j < 3 && i < packed.length; i++, j++, r++, rightShift -= 2) {
            int mask = masks[j][1];
            int leftShift = j * 2;
                
            if (j == 0) {
                result[r]  = (byte)(packed[i] & mask);
            } else {
                result[r] |= (byte)((packed[i] << leftShift) & mask);
            }
    
            if (r < size - 1) {
                mask = masks[j + 1][0];
                result[r + 1] = (byte)((packed[i] >> rightShift) & mask);
            }
        }
        r++; // correction for the last packed byte
    }    
    for (int i = 0; i < result.length; i++) {
        // result[i] += 32;  // restore after unpacking // previous implementation
        if (result[i] < 0x20) {
            result[i] += 0x40; // restore 6 bit for letters
        }
    }
    return result;
}

static String printArr(byte[] arr) {
    return IntStream.range(0, arr.length)
                    .mapToObj(i -> "0x" + Integer.toHexString(arr[i] & 0xFF).toUpperCase())
                    .collect(Collectors.joining(" "));
}

Test:

String text = "ABCDEFGH12";
System.out.println(text);
byte[] bytesFromText = text.getBytes();

byte[] packed = packAscii(bytesFromText);
byte[] unpacked = unpackAscii(packed);

System.out.println(new String(unpacked));

Output

ABCDEFGH12
packed =[-95, 56, -110, -91, 121, -94, -111, 4]
unpacked size=10
ABCDEFGH12

update
Using test word RECIPE 1 for the new mode

RECIPE 1
packed =0x52 0x31 0x24 0x50 0x1 0xC6
unpacked size=8
RECIPE 1

update2
The following implementation supports format mentioned in the question (however, not tested for other sizes of unpacked data than 8)

private static byte[] packAscii(byte[] unpacked) {

    long tmp = 0;
    for (int i = 0, shift = (unpacked.length - 1) * 6; i < unpacked.length; i++, shift -= 6) {
        tmp += ((long)unpacked[i] & 0x3F) << shift;
    }
    
    int size = (int) Math.ceil(unpacked.length * 3.0 / 4.0);
    byte[] result = new byte[size];
    for (int i = size - 1; i >= 0; i--) {
        result[i] = (byte) (tmp & 0xFF);
        tmp >>= 8;
    }
    System.out.println("packed =" + printArr(result));
    return result;
}

private static byte[] unpackAscii(byte[] packed) {
    int size = (int) Math.floor(packed.length * 4.0 / 3.0);
    byte[] result = new byte[size];
    System.out.println("unpacked size=" + size);
    long tmp = 0;
    for (int i = 0; i < packed.length; i++) {
        tmp <<= 8;
        tmp |= ((int)packed[i]) & 0xFF;    
    }

    for (int i = size - 1; i >= 0; i--) {
        result[i] = (byte)(tmp & 0x3F);
        if (result[i] < 0x20)
            result[i] += 0x40;
        tmp >>= 6;
    }
 
    return result;
}

Test succeeds

String text = "RECIPE 1";
System.out.println(text);
byte[] bytesFromText = text.getBytes();

byte[] packed = packAscii(bytesFromText);
byte[] unpacked = unpackAscii(packed);

System.out.println(new String(unpacked));

output

RECIPE 1
packed =0x48 0x50 0xC9 0x40 0x58 0x31
unpacked size=8
RECIPE 1
Related