How to understand the base64 encoding in vtk binary file format

Question

I have a problem with how to understand the binary DataArray. The problem because of the base64 encoding.

The manual said if the format of DataArray is binary,

The data are encoded in base64 and listed contiguously inside the
DataArray element. Data may also be compressed before encoding in base64. The byte-
order of the data matches that specified by the byte_order attribute of the VTKFile element.

I can not fully understand that, so I have obtain ascii file and binary file for same model.

ASCII file



  
    
      
      
      
      
      
        
          0 0 0 1 0 0
          1 1 0 0 1 1
        
      
      
        
          0 1 2 3
        
        
          4
        
        
          10

Binary file



  
    
      
      
      
      
      
        
          AQAAAACAAAAwAAAAEQAAAA==eJxjYEAGDfaobEw+ADwjA7w=
        
      
      
        
          AQAAAACAAAAgAAAAEwAAAA==eJxjYIAARijNBKWZoTQAAHAABw==
        
        
          AQAAAACAAAAIAAAACwAAAA==eJxjYYAAAAAoAAU=
        
        
          AQAAAACAAAABAAAACQAAAA==eJzjAgAACwAL

When I looked at the the DataArray, using the last one as an example, I can not create the relationship between AQAAAACAAAABAAAACQAAAA==eJzjAgAACwAL and 10.

My understanding can be expressed using follow code, but it obtain CggAAA==.

#include "base64.h" // https://github.com/superwills/NibbleAndAHalf/blob/master/NibbleAndAHalf/base64.h
#include 
int main()
{
    int x = 10;
    int len;

    // first arg: binary buffer
    // second arg: length of binary buffer
    // third arg: length of ascii buffer
    char *ascii = base64((char *)&x, sizeof(int), &len);
    
    std::cout << ascii << std::endl;
    std::cout << len << std::endl;
    free(ascii);
    return 0;
}

Can someone give me an explanation of how to convert? Another relate topic can be see in

https://discourse.vtk.org/t/error-when-writing-binary-vtk-files/4487/7

Thanks for your time.

user_10877953 · Answer

Solution can be found in disscusstion.

https://discourse.vtk.org/t/how-to-understand-binary-dataarray-in-xml-vtk-output/4489

The long extra data comes from the compressor header.

user_13132315 · Answer

I have found the solution and written the answer in a VTK support question, but I am writing it here in case anyone comes here looking for the same issue as us two.

Note that I program in Python, but I believe there are base64 and zlib functions in C++. Also, I use numpy to define arrays, but I believe std::vector can be equivalently used in C++.

So, suppose we want to write the single precision float32 array called "Points" in your example. If we suppose that a header type of "UInt32" is used, then in Python, we would do:

import numpy as np
import zlib
import base64

# write the float array.
arr = np.array([0, 0, 0, 1, 0, 0,
                1, 1, 0, 0, 1, 1], dtype='float32')
# generate a zlib compressed array. This outputs a python byte type
arr_comp = zlib.compress(arr)

# generate the uncompressed header
header = np.array([ 1,  # apparently this is always the case, I think
                2**15,  # from what I have read, this is true in general
           arr.nbytes,  # the size of the array `arr` in bytes
       len(arr_comp)],  # the size of the compressed array
                  dtype='uint32')  # because of header_type="UInt32"

# use base64 encoding when writing to file
# `.decode("utf-8")` transforms the python byte type to a string
print((base64.b64encode(header_arr) + base64.b64encode(arr_comp)).decode("utf-8"))

The output is as expected:

AQAAAACAAAAwAAAAEQAAAA==eJxjYEAGDfaobEw+ADwjA7w=

The 2**15 is the argument that controls the size of the history buffer (or the “window size”) used when compressing data, according to the zlib python docs. Not sure what that means though...

Edit: The above code only works if the size in bytes of the array is less than or equal to 2**15. In the VTK support question I have expanded for the case where the array is larger. You have to divide it into chunks.

How to understand the base64 encoding in vtk binary file format

ASCII file

Binary file

2 Answers