How can I use serde_json to deserialize arbitrary JSON into a Value-like object of raw bytes?

I'm writing a library to deserialize a subset of JSON into predefined Python types.

I want to deserialize arbitrary JSON into an object that quacks like serde-json's Value. However, I don't want it to deserialize into String's, Number's and Bool's - instead when the deserializer hits one of these I would prefer it simply keeps a reference to the respective byte string so I can efficiently (i.e. without the additional type conversion) parse the byte strings into the correct arbitrary Python types. Something like this:

use serde::Deserialize;
use serde_json::value::RawValue;
use serde_json::Map;

#[derive(Deserialize)]
pub enum MyValue<'a> {
    Null,
    Bytes(&'a RawValue),
    Array(Vec<MyValue<'a>>),
    Object(Map<String, MyValue<'a>>),
}

This will require writing a lot of traits so that it behaves like Value, and I'm not even sure if it won't just ignore deserializing the structural parts and put everything into a RawValue.

What is the cleanest way to do this?