What is the proper way to search a file byte-by-byte?

In parsing a file, I need to locate a certain valued byte. In this case, I am searching for the value 13. I know that this value will exist at the start of a 32-byte block. With all of that information, I have come up with a few solutions similar to the following:

let mut file = File::open("file_to_open").unwrap();
let mut read_array = [0u8;32];
while read_array[0] != 13 {
    file.read_exact(&mut read_array).unwrap();
}
[...]

The problem with this is it takes a very long time. Running this code 100,000 times, for instance, takes around 9 seconds. On the other hand, if I load a large amount of data into a Vec and then search through memory, I can read it much faster. But, the file is large and if I load too big of a chunk at a time, it will take longer than reading byte-by-byte in cases where the data would have been found early in the file. I tried using SeekFrom::Current, but seeking forward 32 bytes was slower than just reading all 32 bytes at once.

Is my only option to come up with an acceptable chunk size and iterate those chunks?

Edit:

The following is currently my quickest implementation:

fn get_raw(file: &mut File) -> usize {
    let start_bit= 32;
    let mut iter = 0;
    file.seek(SeekFrom::Start(start_bit)).unwrap();
    let read_size = 32;
    let block_size = read_size * 100;
    let mut my_vec = vec![0u8;block_size];
    while let Ok(n) = file.read(&mut my_vec) {
        if n != block_size {
            break;
        }
        for i in (0..block_size).step_by(read_size) {
            if my_vec[i] == 13 {
                return iter;
            }
            iter += 1;
        }
    }
    panic!("The database is corrupt; this process cannot continue.");
}