During machine learning course, the professor asks us to implement, using SVM, a classifier for handwritten digits. He gave us this dataset, that contains also a train set to test the classifier.
I tried to write a java class for reading the dataset and converting it into a data structure (an HashMap, for example), but I encountered a lot of difficulties while I read the files. On the dataset's website, there is the structure of this IDX file:
The basic format is:
magic number
size in dimension 0
size in dimension 1
size in dimension 2
.....
size in dimension N
data
The magic number is an integer (MSB first). The first 2 bytes are always 0.
The third byte codes the type of the data:
0x08: unsigned byte
0x09: signed byte
0x0B: short (2 bytes)
0x0C: int (4 bytes)
0x0D: float (4 bytes)
0x0E: double (8 bytes)
The 4-th byte codes the number of dimensions of the vector/matrix: 1 for vectors, 2 for matrices....
The sizes in each dimension are 4-byte integers (MSB first, high endian, like in most non-Intel processors).
The data is stored like in a C array, i.e. the index in the last dimension changes the fastest.
So I started with the file t10k-labels-idx1-ubyte:
[offset] [type] [value] [description]
0000 32 bit integer 0x00000801(2049) magic number (MSB first)
0004 32 bit integer 10000 number of items
0008 unsigned byte ?? label
0009 unsigned byte ?? label
........
xxxx unsigned byte ?? label
This is its structure, so I wrote a java class that should read the file and compute the magic number and the number of items. This is the class:
public class ReadTest {
static String testLabel = "t10k-labels-idx1-ubyte";
public static void main(String[] args) {
byte[] bytes = null;
byte[] fileName = testLabel.getBytes(StandardCharsets.UTF_8);
try {bytes = Files.readAllBytes(Paths.get(testLabel));}
catch (IOException e) {System.err.println("Error: " + e.getMessage());}
if(bytes != null) {
List<Byte> listBytes = new ArrayList<>();
List<Byte> listName = new ArrayList<>();
for(int i=0; i<bytes.length; i++) listBytes.add(bytes[i]);
for(int i=0; i<fileName.length; i++) listName.add(fileName[i]);
System.out.println("I read " + listBytes.size() + " bytes!");
System.out.println("Filename: " + listName.size() + " bytes!");
if(listBytes.containsAll(listN)) System.out.println("Gotcha!");
byte[] magic = {bytes[3], bytes[2], bytes[1], bytes[0]};
byte[] items = {bytes[7], bytes[6], bytes[5], bytes[4]};
System.out.println("[REVERSE] Magic: " + ByteBuffer.wrap(magic).getInt() + ". Items: " + ByteBuffer.wrap(items).getInt());
System.out.println("[AFTER 32] Magic: " + ByteBuffer.wrap(bytes, 32, 4).getInt() + ". Items: " + ByteBuffer.wrap(bytes, 36, 4).getInt());
System.out.println("[RAW] Magic: " + ByteBuffer.wrap(bytes, 0, 4).getInt() + ". Items: " + ByteBuffer.wrap(bytes, 4, 4).getInt());
}
}
}
I was expecting, as output, "Magic: 2049. Items: 10000", but it was not. At the begin I only read the first 8 bytes (the [RAW] print), then I tried to read them in reverse order (the [REVERSE] print). After that, I ran an hexdump on the file trying to understand where was the problem: if I didn't understand it wrongly, in the IDX file there is an header containing some information about the file, including the name. So, I converted the file name into a byte array and, using two List, I found out I was right (hopefully). So I tried, skipping the first 32 bytes, to compute again the two numbers but without success.
Did I make mistakes in the implementation? Are there some classes that help me to read these files correctly?
Sorry for the looong post, but I tried to explain the entire process (: