Store a reference to the underlying buffer from a struct that doesn't live long enough

user_8181162 0
Viewed 121

I'm attempting to write an incremental XML parser in Rust using quick_xml.

Some of the XML files will not fit in memory (on my laptop) so I'm trying to only store relevant chunks of each file in a buffer of Vec<u8>.

Within each file chunk of Vec<u8> I want to store borrows to slices in some struct Data

quick_xml provides a read_event method which appends to the buffer and returns a quick_xml::events::Event (an enum containing a struct with a buf: Cow<'a, [u8]> field which borrows from the buffer)

Essentially I want to take the data referenced by the Event and store it in my Data struct.

However the borrow checker has a heart attack because the Event only lives for the call to read_event and I'm trying to keep a reference to it that lives as long as the data in the buffer.

The code below is the implementation of what I have tried to describe above. Could I get some help in storing a borrow to the underlying buf from an Event?

use quick_xml::events::Event;
use quick_xml::Reader;

const XML: &str = r#"<?xml version="1.0" encoding="UTF-8"?>
<RUN_SET xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <RUN xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" alias="HAP1 gene trap unselected control dataset" accession="SRR2034585" center_name="Stanford University">
    <IDENTIFIERS>
      <PRIMARY_ID>SRR2034585</PRIMARY_ID>
      <SUBMITTER_ID namespace="Stanford University">HAP1 gene trap unselected control dataset</SUBMITTER_ID>
    </IDENTIFIERS>
    <EXPERIMENT_REF accession="SRX1034759"/>
  </RUN>
</RUN_SET>
"#;

#[derive(Debug)]
struct Data<'a> {
    primary_id: Option<&'a [u8]>,
    experiment_ref: Option<&'a [u8]>,
}


fn main() {
    let mut buf: Vec<u8> = vec![];
    let mut reader = Reader::from_str(XML);
    let mut depth = 0;
    let mut path: Vec<u8> = vec![];
    reader.expand_empty_elements(true);
    let mut data = Data { primary_id: None, experiment_ref: None };
    loop {
        match reader.read_event(&mut buf) {
            Ok(Event::Start(ref e)) => {
                depth += 1;
                path.push(b"/"[0]);
                path.append(&mut e.name().to_vec());

                if path == "/RUN_SET/RUN/EXPERIMENT_REF".as_bytes() {
                    let experiment_ref = // What to put here?
                    data = Data { experiment_ref, ..data };
                }
            }
            Ok(Event::End(ref e)) => {
                depth -= 1;
                path.truncate(path.len() - e.name().len() - 1);
            }
            Ok(Event::Eof) => { break; }
            _ => {}
        }
        if depth == 1 {
            println!("{:?}", data);
            buf.clear();
            path.clear();
        }
    }
    
}
borrow-checker rust reference
1 Answers

Calling read_event will cause the buffer to expand if necessary, which can change its address, so any references become invalid. Specifically, you are trying to call read_event, store a reference (data) pointing into the buffer, then call read_event again which can move the buffer.

It seems the best way to solve this is to move/clone the event name so that its lifetime is not bound to the buffer. Frustratingly, it seems that quick_xml::events::BytesStart<'a> exposes no way to directly move the underlying Cow<'a, [u8]> so we have to store the BytesStart object itself in order to avoid a potentially unnecessary clone.

Here is one way to do this. I made significant changes to the code in order to more accurately/efficiently do what I think you intended:

use quick_xml::events::Event;
use quick_xml::Reader;

const XML: &str = r#"<?xml version="1.0" encoding="UTF-8"?>
<RUN_SET xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <RUN xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" alias="HAP1 gene trap unselected control dataset" accession="SRR2034585" center_name="Stanford University">
    <IDENTIFIERS>
      <PRIMARY_ID>SRR2034585</PRIMARY_ID>
      <SUBMITTER_ID namespace="Stanford University">HAP1 gene trap unselected control dataset</SUBMITTER_ID>
    </IDENTIFIERS>
    <EXPERIMENT_REF accession="SRX1034759"/>
  </RUN>
</RUN_SET>
"#;

#[derive(Debug)]
struct Data<'a> {
    primary_id: Option<&'a [u8]>,
    experiment_ref: Option<quick_xml::events::BytesStart<'static>>,
}

fn main() {
    let target: &[&[u8]] = &[b"RUN_SET", b"RUN", b"EXPERIMENT_REF"];
    let mut buf: Vec<u8> = vec![];
    let mut reader = Reader::from_str(XML);
    let mut depth = 0;
    let mut good = 0;
    reader.expand_empty_elements(true);
    let mut data = Data {
        primary_id: None,
        experiment_ref: None,
    };
    loop {
        match reader.read_event(&mut buf) {
            Ok(Event::Start(e)) => {
                if depth == good && target.get(depth) == Some(&e.name()) {
                    good += 1;
                    if good == target.len() {
                        data = Data {
                            experiment_ref: Some(e.into_owned()),
                            ..data
                        };
                    }
                }
                depth += 1;
            }
            Ok(Event::End(_)) => {
                depth -= 1;
                good = good.min(depth);
            }
            Ok(Event::Eof) => {
                buf.clear();
                break;
            }
            _ => {}
        }
        buf.clear();
    }
    println!("{:?}", data);
}
user_33988390
Related