Read binary file in units of f64 in Rust

user_7746905 0
Viewed 798

Assuming you have a binary file example.bin and you want to read that file in units of f64, i.e. the first 8 bytes give a float, the next 8 bytes give a number, etc. (assuming you know endianess) How can this be done in Rust?

I know that one can use std::fs::read("example.bin") to get a Vec<u8> of the data, but then you have to do quite a bit of "gymnastics" to convert always 8 of the bytes to a f64, i.e.

fn eight_bytes_to_array(barry: &[u8]) -> &[u8; 8] {
    barry.try_into().expect("slice with incorrect length")
}

let mut file_content = std::fs::read("example.bin").expect("Could not read file!");
let nr = eight_bytes_to_array(&file_content[0..8]);
let nr = f64::from_be_bytes(*nr_dp_per_spectrum);

I saw this post, but its from 2015 and a lot of changes have happend in Rust since then, so I was wondering if there is a better/faster way these days?

rust file-processing
2 Answers

Example without proper error handling and checking for cases when file contains not divisible amount of bytes.

use std::fs::File;
use std::io::{BufReader, Read};

fn main() {
    // Using BufReader because files in std is unbuffered by default
    // And reading by 8 bytes is really bad idea.
    let mut input = BufReader::new(
        File::open("floats.bin")
        .expect("Failed to open file")
    );

    let mut floats = Vec::new();
    loop {
        use std::io::ErrorKind;
        // You may use 8 instead of `size_of` but size_of is less error-prone.
        let mut buffer = [0u8; std::mem::size_of::<f64>()];
        // Using read_exact because `read` may return less 
        // than 8 bytes even if there are bytes in the file.
        // This, however, prevents us from handling cases
        // when file size cannot be divided by 8.
        let res = input.read_exact(&mut buffer);
        match res {
            // We detect if we read until the end.
            // If there were some excess bytes after last read, they are lost.
            Err(error) if error.kind() == ErrorKind::UnexpectedEof => break,
            // Add more cases of errors you want to handle.
            _ => {}
        }
        // You should do better error-handling probably.
        // This simply panics.
        res.expect("Unexpected error during read");
        // Use `from_be_bytes` if numbers in file is big-endian
        let f = f64::from_le_bytes(buffer);
        floats.push(f);
    }
}
user_81959870

I would create a generic iterator that returns f64 for flexibility and reusability.

struct F64Reader<R: io::BufRead> {
    inner: R,
}

impl<R: io::BufRead> F64Reader<R> {
    pub fn new(inner: R) -> Self {
        Self{
            inner
        }
    }
}

impl<R: io::BufRead> Iterator for F64Reader<R> {
    type Item = f64;

    fn next(&mut self) -> Option<Self::Item> {
        let mut buff: [u8; 8] = [0;8];
        self.inner.read_exact(&mut buff).ok()?;
        Some(f64::from_be_bytes(buff))
    }
}

This means if the file is large, you can loop through the values without storing it all in memory

let input = fs::File::open("example.bin")?;
for f in F64Reader::new(io::BufReader::new(input)) {
    println!("{}", f)
}

Or if you want all the values you can collect them

let input = fs::File::open("example.bin")?;
let values : Vec<f64> = F64Reader::new(io::BufReader::new(input)).collect();
user_26917590
Related