Deserialize JSON list of hex strings as bytes

user_27302 0
Viewed 629

Iʼm trying to read a JSON stream, part of which looks like

  "data": [
    "c1a8f800a4393e0cacd05a5bc60ae3e0",
    "bbac4013c1ca3482155b584d35dac185",
    "685f237d4fcbd191c981b94ef6986cde",
    "a08898e81f1ddb6612aa12641b856aa9"
  ]

(there are more entries in the data list and each each entry is longer, but this should be illustrative; both the length of the list and the length of each hex string is known at compile time)

Ideally Iʼd want a single [u8; 64] (the actual size is known at compile time), or failing that, a Vec<u8>, but I imagine itʼs easier to deseriazie it as a Vec<[u8; 16]> and merge them later. However, Iʼm having trouble doing even that.

The hex crate has a way to deserialize a single hex string as a Vec or array of u8, but I canʼt figure out how to tell Serde to do that for each entry of the list. Is there a simple way to do that Iʼm overlooking, or do I need to write my own list deserializer?

serde rust
2 Answers

Serde has the power to use serializers and deserializers from other crates in a nested fashion using #[serde(with = "...")]. Since hex has a serde feature, this can be easily done.

Here is a simple example using serde_json and hex.

cargo.toml

serde = { version =  "1.0.133",  features = ["derive"] }
serde_json = "1.0.74"
hex = { version = "0.4", features = ["serde"] }

main.rs

use serde::{Deserialize, Serialize};
use serde_json::Result;

#[derive(Serialize, Deserialize, Debug)]
struct MyData {
    data: Vec<MyHex>,
}

#[derive(Serialize, Deserialize, Debug)]
#[serde(transparent)]
struct MyHex {
    #[serde(with = "hex::serde")]
    hex: Vec<u8>,
}


fn main() -> Result<()> {
    let data = r#"
    {
        "data": [
            "c1a8f800a4393e0cacd05a5bc60ae3e0",
            "bbac4013c1ca3482155b584d35dac185",
            "685f237d4fcbd191c981b94ef6986cde",
            "a08898e81f1ddb6612aa12641b856aa9"
        ]
    }
    "#';

    let my_data: MyData = serde_json::from_str(data)?;

    println!("{:?}", my_data); // MyData { data: [MyHex { hex: [193, 168, 248, 0, 164, 57, 62, 12, 172, 208, 90, 91, 198, 10, 227, 224] }, MyHex { hex: [187, 172, 64, 19, 193, 202, 52, 130, 21, 91, 88, 77, 53, 218, 193, 133] }, MyHex { hex: [104, 95, 35, 125, 79, 203, 209, 145, 201, 129, 185, 78, 246, 152, 108, 222] }, MyHex { hex: [160, 136, 152, 232, 31, 29, 219, 102, 18, 170, 18, 100, 27, 133, 106, 169] }] }


    return Ok(());
}

Serde With Reference
Hex Serde Reference

user_178340010

In some performance-critical situations, it may be advantageous to implement your own deserializer and use it with serde(deserialize_with = …).

If you go that route, you have to:

  • Implement a deserialziation function for data
  • Implement a visitor which takes a sequence of precisely 4 blocks
  • These blocks then need another deserialization function that turns a string into [u8; 16]
use serde::{Deserialize, Deserializer};

#[derive(Deserialize, Debug)]
pub struct Foo {
    #[serde(deserialize_with = "deserialize_array_of_hex")]
    pub data: [u8; 64],
}

fn deserialize_array_of_hex<'de, D: Deserializer<'de>>(d: D) -> Result<[u8; 64], D::Error> {
    use serde::de;
    use std::fmt;

    #[derive(serde_derive::Deserialize)]
    struct Block(#[serde(with="hex::serde")] [u8; 16]);

    struct VecVisitor;
    impl<'de> de::Visitor<'de> for VecVisitor {
        type Value = [u8; 64];

        fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
            write!(formatter, "a list containing 4 hex strings")
        }

        fn visit_seq<A>(self, mut seq: A) -> Result<Self::Value, A::Error>
        where
            A: de::SeqAccess<'de>,
        {
            let mut data = [0; 64];
            for i in 0..4 {
                let block = seq
                    .next_element::<Block>()?
                    .ok_or_else(|| de::Error::custom("too short"))?;
                for j in 0..16 {
                    data[i * 16 + j] = block.0[j];
                }
            }
            if seq.next_element::<String>()?.is_some() {
                return Err(de::Error::custom("too long"))
            }
            Ok(data)
        }
    }

    d.deserialize_seq(VecVisitor)
}

Full example playground. One could also implement DeserializeSeed for Block and only pass a reference to the [u8; 64] to be written into, but I suspect that copying 16 bytes is negligibly cheap. (Edit: I measured it, it turns out about 10% faster than the other two solutions in this post (when using hex::decode_to_slice in visit_str).)

Actually, nevermind having to implement your own deserializer for performance, the above solution is about equal in performance to

use serde::Deserialize;

#[derive(Deserialize, Debug)]
#[serde(from = "MyDataPre")]
pub struct MyData {
    pub data: [u8; 64],
}

impl From<MyDataPre> for MyData {
    fn from(p: MyDataPre) -> Self {
        let mut data = [0; 64];
        for b in 0..4 {
            for j in 0..16 {
                data[b * 16 + j] = p.data[b].0[j];
            }
        }
        MyData { data }
    }
}

#[derive(Deserialize, Debug)]
pub struct MyDataPre {
    data: [MyHex; 4],
}

#[derive(Deserialize, Debug)]
struct MyHex (#[serde(with = "hex::serde")] [u8; 16]);

The trick here is the use of #[serde(from = …)], which allows you to deserialize to some other struct, and then tell serde how to convert that to the struct you originally wanted.

user_4010590
Related