How do I read CSV data without knowing the structure at compile time?

user_14455289 0
Viewed 997

I'm pretty new to Rust and trying to implement some kind of database. Users should create tables by giving a table name, a vector of column names and a vector of column types (realized over an enum). Filling tables should be done by specifying csv files. However, this requires the structure of the table rows to be specified at compile time, like shown in the basic example:

#[derive(Debug, Deserialize, Eq, PartialEq)]
struct Row {
    key: u32,
    name: String,
    comment: String
}
use std::error::Error;
use csv::ReaderBuilder;
use serde::Deserialize;
use std::fs;

fn read_from_file(path: &str) -> Result<(), Box<dyn Error>> {
    let data = fs::read_to_string(path).expect("Unable to read file");
    let mut rdr = ReaderBuilder::new()
        .has_headers(false)
        .delimiter(b'|')
        .from_reader(data.as_bytes());
    let mut iter = rdr.deserialize();

    if let Some(result) = iter.next() {
        let record:Row = result?;
        println!("{:?}", record);
        Ok(())
    } else {
        Err(From::from("expected at least one record but got none"))
    }   
}

Is there a possibility to use the generic table information instead of the "Row"-struct to cast the results from the deserialization? Is it possible to simply allocate memory according to the combined sizes of the column types and parse the records in? I would do something like this in C...

rust deserialization
1 Answers

Is there a possibility to use the generic table information instead of the "Row"-struct to cast the results from the deserialization?

All generics replaced with concrete types at compile time. If you do not know types you will need in runtime, "generics" is not what you need.

Is it possible to simply allocate memory according to the combined sizes of the column types and parse the records in? I would do something like this in C...

I suggest using Box<dyn Any> instead, to be able to store reference of any type and, still, know what type it is.

Maintenance cost for this approach is pretty high. You have to manage each possible value type everywhere you want to use a cell's value. On the other hand, you do not need to parse value each time, just make some type checks in runtime.

I have used std::any::TypeId to identify type, but it can not be used in match expressions. You can consider using custom enum as type identifier.

use std::any::{Any, TypeId};
use std::io::Read;

use csv::Reader;

#[derive(Default)]
struct Table {
    name: String,
    headers: Vec<(String, TypeId)>,
    data: Vec<Vec<Box<dyn Any>>>,
}

impl Table {
    fn add_header(&mut self, header: String, _type: TypeId) {
        self.headers.push((header, _type));
    }

    fn populate_data<R: Read>(
        &mut self,
        rdr: &mut Reader<R>,
    ) -> Result<(), Box<dyn std::error::Error>> {
        for record in rdr.records() {
            let record = record?;
            let mut row: Vec<Box<dyn Any>> = vec![];
            for (&(_, type_id), value) in self.headers.iter().zip(record.iter()) {
                if type_id == TypeId::of::<u32>() {
                    row.push(Box::new(value.parse::<u32>()?));
                } else if type_id == TypeId::of::<String>() {
                    row.push(Box::new(value.to_owned()));
                }
            }
            self.data.push(row);
        }
        Ok(())
    }
}

impl std::fmt::Display for Table {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        writeln!(f, "Table: {}", self.name)?;
        for (name, _) in self.headers.iter() {
            write!(f, "{}, ", name)?;
        }
        writeln!(f)?;
        for row in self.data.iter() {
            for cell in row.iter() {
                if let Some(&value) = cell.downcast_ref::<u32>() {
                    write!(f, "{}, ", value)?;
                } else if let Some(value) = cell.downcast_ref::<String>() {
                    write!(f, "{}, ", value)?;
                }
            }
            writeln!(f)?;
        }
        Ok(())
    }
}

fn main() {
    let mut table: Table = Default::default();
    table.name = "Foo".to_owned();
    table.add_header("key".to_owned(), TypeId::of::<u32>());
    table.add_header("name".to_owned(), TypeId::of::<String>());
    table.add_header("comment".to_owned(), TypeId::of::<String>());
    let data = "\
key,name,comment
1,foo,foo comment
2,bar,bar comment
";
    let mut rdr = Reader::from_reader(data.as_bytes());
    table.populate_data(&mut rdr).unwrap();
    print!("{}", table);
}
user_8333980
Related