how to read in chunks using tokio::io::BufReader

Viewed 2646

I am calculating the checksum of a file using something like this:

pub fn sha256_digest(file_path: &str) -> Result<String, Box<dyn Error>> {
    let file = fs::File::open(file_path)?;
    let mut reader = BufReader::new(file);
    let mut context = Context::new(&SHA256);

    loop {
        let consummed = {
            let buffer = reader.fill_buf()?;
            if buffer.is_empty() {
                break;
            }
            context.update(buffer);
            buffer.len()
        };
        reader.consume(consummed);
    }

    let digest = context.finish();

    Ok(write_hex_bytes(digest.as_ref()))
}

Checksum:

dca3b9746da896f05072bdec6b788513029b26ab453b82e2e9d4365e56e2c913
Elapsed: 226.14ms

I tried already this with tokio:

async fn sha256_digest(file_path: &str) -> Result<String, Box<dyn Error>> {
    let file = File::open(file_path).await?;
    let mut stream = FramedRead::new(file, BytesCodec::new());
    let mut context = Context::new(&SHA256);
    while let Some(bytes) = stream.try_next().await? {
        context.update(&bytes);
    }
    let digest = context.finish();
    Ok(write_hex_bytes(digest.as_ref()))
}

But unfortunately, it takes like twice the time to calculate the checksums, therefore I would like to try an option with tokio::io::BufReader the problem that I am having is to implement the Trait AsyncBufRead so that I could use the consume method.

Checksum:

dca3b9746da896f05072bdec6b788513029b26ab453b82e2e9d4365e56e2c913
Elapsed: 448.25ms

I am trying:

async fn sha256_digest_BufReader(file_path: &str) -> Result<String, Box<dyn Error>> {
    let file = File::open(file_path).await?;
    let mut reader = BufReader::new(file);
    let mut context = Context::new(&SHA256);
    loop {
        let consummed = {
            let buffer = reader.buffer();
            if buffer.is_empty() {
                break;
            }
            context.update(buffer);
            buffer.len()
        };
        reader.consume(consummed);
    }
    let digest = context.finish();
    Ok(write_hex_bytes(digest.as_ref()))
}

But getting this error:

error[E0599]: no method named `consume` found for struct `tokio::io::util::buf_reader::BufReader<tokio::fs::file::File>` in the current scope
  --> src/main.rs:63:16
   |
63 |         reader.consume(consummed);
   |                ^^^^^^^ method not found in `tokio::io::util::buf_reader::BufReader<tokio::fs::file::File>`

warning: unused import: `tokio::io::AsyncBufRead`

Any idea of how could this be implemented?

Update: I am trying this:

async fn sha256_digest_BufReader(file_path: &str) -> Result<String, Box<dyn Error>> {
    let file = File::open(file_path).await?;
    let mut reader = BufReader::new(file);
    let mut context = Context::new(&SHA256);
    let mut buf: [u8; 8192] = [0; 8192]; //chunk size (8K, 65536, etc)

    while let Ok(size) = reader.read(&mut buf[..]).await {
        if size == 0 {
            break;
        }
        context.update(&buf[0..size]);
    }
    let digest = context.finish();
    Ok(write_hex_bytes(digest.as_ref()))
}

But still running "slow" or at least twice the time even changing the chunk size like for example testing with 65536, 131072.

0 Answers
Related