unpacking binary file via octets->string->unpack fails: signed int `#(243 0)` is illegal UTF8

user_1031776 0
Viewed 101

I am parsing a binary file (nifti) with a mix of chars, floats, ints, and shorts (using the PDL::IO::Nifti cpan module as reference).

I am having some luck parsing sequences of octets to a string so they can be passed to cl-pack:unpack. This is convoluted but convenient for porting using the perl module as reference.

This strategy fails on reading #(243 0) as binary

(setf my-problem (make-array 2
                             :element-type '(unsigned-byte 8)
                             :initial-contents #(243 0)))
(babel:octets-to-string my-problem)

Illegal :UTF-8 character starting at position 0

and, when trying to read the file as char*

the octet sequence #(243 0 1 0) cannot be decoded.

I'm hoping there is a simple encoding issue I haven't figured out. Trying to go in the reverse direction (packing 243 and getting octets) gives a vector of length 3 for what I expect to be 2.

(babel:string-to-octets (cl-pack:pack "s" 243))
; yields #(195 179 0) expect #(243 0)

Full context

;; can read up to position 40. at which we expect 8 signed ints. 
;; 4th int is value "243" but octet cannot be parsed
(setq fid-bin (open "test.nii" :direction :input :element-type 'unsigned-byte))
(file-position fid-bin 40)
(setf seq (make-array (* 2 8) :element-type '(unsigned-byte 8)))
(read-sequence seq fid-bin) 
; seq: #(3 0 0 1 44 1 243 0 1 0 1 0 1 0 1 0)

(babel:octets-to-string seq) ; Illegal :UTF-8 character starting at position 6.
(sb-ext:octets-to-string seq) ; Illegal ....

;; first 3 are as expected
(cl-pack:unpack "s3" (babel:octets-to-string (subseq seq 0 6)))
; 3 256 300

(setf my-problem (subseq seq 6 8)) ; #(243 0)
(babel:octets-to-string my-problem)       ; Illegal :UTF-8 character starting at position 0.

;; checking the reverse direction
;; 243 gets represented as 3 bytes!?
(babel:string-to-octets (cl-pack:pack "s3" 3 256 300))     ; #(3 0 0 1 44 1)
(babel:string-to-octets (cl-pack:pack "s4" 3 256 300 243)) ; #(3 0 0 1 44 1 195 179 0)


(setq fid-str (open "test.nii" :direction :input))
(setf char-seq (make-array (* 2 8) :initial-element nil :element-type 'char*))
(file-position fid-str 40)
(read-sequence char-seq fid-str)
;; :UTF-8 stream decoding error on #<SB-SYS:FD-STREAM ....
;; the octet sequence #(243 0 1 0) cannot be decoded.

The perl equivalent

open my $f, "test.nii";
seek $f, 46, 0;
read $f,my $b, 2;
print(unpack "s", $b); # 243
utf-8 common-lisp binaryfiles
2 Answers

The problem is that you are using functions which try to treat some sequence of octets as a representation of an encoding of a sequence of characters (or of some Unicode things: I think there are things other than characters in Unicode). In particular, in your case, the functions you are using are treating a sequence of octets as the UTF-8 encoding of some string. Well, not all sequences of octets are legal UTF-8 so the functions are, correctly, puking on an illegal sequence of octets.

But that's because you're not doing the right thing: what you want to do is to take a sequence of octets and make a string whose char-codes are those octets. You don't want to be doing with any silly encoding-big-characters-in-small-integers rubbish, because you will never see any big characters. You want something like these functions (both somewhat misnamed, since they aren't fussed about the whole octet thing unless you are).

(defun stringify-octets (octets &key 
                                (element-type 'character)
                                (into (make-string (length octets)
                                                   :element-type element-type)))
  ;; Smash a sequence of octets into a string.
  (map-into into #'code-char octets))

(defun octetify-string (string &key
                               (element-type `(integer 0 (,char-code-limit)))
                               (into (make-array (length string)
                                                 :element-type element-type)))
  ;; smash a string into an array of 'octets' (not actually octets)
  (map-into into #'char-code string))

And now you can check everything works:

> (octetify-string (pack "s" 243))
#(243 0)

>  (unpack "s" (stringify-octets (octetify-string (pack "s" 243))))
243

and so on. Given your example sequence:

> (unpack "s8" (stringify-octets #(3 0 0 1 44 1 243 0 1 0 1 0 1 0 1 0)))
3
256
300
243
1
1
1
1

A really much better approach would be to have the packing & unpacking functions simply handle sequences of octets. But I suspect that's a lost cause. An interim approach which is horrible but less horrible than converting sequences of octets to characters would be to read the file as text but with an external-format which does no translation at all. How to do that is implementation-dependent (but something based on latin-1 will be a good start).

Admin1

It seems that the problem is indeed encoding-related:

CL-USER> (cl-pack:pack "s" 243)
"ó\0"

which is the same as the result of:

(babel:octets-to-string my-problem :encoding :iso-8859-1)
user_9770520
Related