Serde MsgPack versioning

user_15602349 0
Viewed 207

I need to serialize some data into files. For the sake of memory efficiency, I want to use the default compact serializer of MessagePack (MsgPack), as it only serializes field values w/o their names. I also want to be able to make changes to the data structure in future versions, which obviously can't be done w/o also storing some meta/versioning information. I imagine the most efficient way to do it is to simply use some "header" field for that purpose. Here is an example:

pub struct Data {
    pub version: u8,
    pub items: Vec<Item>,
}
pub struct Item {
    pub field_a: i32,
    pub field_b: String,
    pub field_c: i16,  // Added in version 3
}

Can I do something like that in rmp-serde (or maybe some other crate?) - to somehow annotate that a certain struct field should only be taken into account for specific file versions?

msgpack serde rust
1 Answers

You can achieve this by writing a custom deserializer like this:

use serde::de::Error;
use serde::{Deserialize, Deserializer, Serialize};

#[derive(Serialize)]
pub struct Data {
    pub version: u8,
    pub items: Vec<Item>,
}

#[derive(Serialize)]
pub struct Item {
    pub field_a: i32,
    pub field_b: String,
    pub field_c: i16, // Added in version 3
}

impl<'de> Deserialize<'de> for Data {
    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
    where
        D: Deserializer<'de>,
    {
        // Inner structs, used for deserializing only
        #[derive(Deserialize)]
        pub struct InnerData {
            version: u8,
            items: Vec<InnerItem>,
        }
        #[derive(Deserialize)]
        pub struct InnerItem {
            field_a: i32,
            field_b: String,
            field_c: Option<i16>, // Added in version 3 - note that this field is optional
        }

        // Deserializer the inner structs
        let inner_data = InnerData::deserialize(deserializer)?;        
        
        // Get the version so we can add custom logic based on the version later on
        let version = inner_data.version;

        // Map the InnerData/InnerItem structs to Data/Item using our version based logic
        Ok(Data {
            version,
            items: inner_data
                .items
                .into_iter()
                .map(|item| {
                    Ok(Item {
                        field_a: item.field_a,
                        field_b: item.field_b,
                        field_c: if version < 3 {
                            42 // Default value
                        } else {
                            // Get the value of field_c
                            // If it's missing return an error, since it's required since version 3
                            // Otherwise return the value
                            item.field_c
                                .map_or(Err(D::Error::missing_field("field_c")), Ok)?
                        },
                    })
                })
                .collect::<Result<_, _>>()?,
        })
    }
}

Short explanation how the deserializer works:

  • We create a "dumb" inner struct which is a copy of your structs but the "new" fields are optional
  • We deserialize to the new inner structs
  • We map from our inner to our outer structs using version-based logic
    • If one of the new fields is missing in a new version we return a D::Error::missing_field error
user_32579010
Related