Rust Borrrow Checker - passing a mutable variable as mutable to a fn and get ownership back

user_1183252 0
Viewed 52

I am fiddling with writing a Lexer that takes an input &str and returns an Iterator over the tokens I generate for the input:

pub enum Token<'a> {
    Lparen(&'a str),         // (
    Rparen(&'a str),         // )
    Lbrack(&'a str),         // [

    // -- snip --
}

pub fn tokenize(input: &str) -> impl Iterator<Item = Token> {
    let mut chars = input.char_indices();

    std::iter::from_fn(move || {

        if let Some(next) = chars.next() {

            let (start, c) = next;
            let end = start + 1;

            return match c {
                '(' => Some(Token::Lparen(&input[start..end])),
                ')' => Some(Token::Rparen(&input[start..end])),
                '[' => Some(Token::Lbrack(&input[start..end])),

                '='  => from_eq(input, start, chars),
                // move of chars occurs here  ^^^^^

                // -- snip --

                _ => Some(Token::Unknown(&input[start..end])),
            }

        }

        None
    })
}

fn from_eq<'a>(input: &'a str, start: usize, mut chars: CharIndices) -> Option<Token<'a>> {
    // -- snip --
}

I am moving chars into the closure and within the closure I pass it on to from_eq. from_eq is supposed to iterate further on chars with a specific handling of the characters it finds.

Now of course, I the borrow checker nags at me

error[E0507]: cannot move out of `chars`, a captured variable in an `FnMut` closure
   --> src/lib.rs:106:47
    |
94  |       let mut chars = input.char_indices();
    |           --------- captured outer variable
95  | 
96  |       std::iter::from_fn(move || {
    |  ________________________-
97  | |
98  | |         if let Some(next) = chars.next() {
99  | |
...   |
106 | |                 '='  => from_eq(input, start, chars),
    | |                                               ^^^^^ move occurs because `chars` has type `CharIndices<'_>`, which does not implement the `Copy` trait
...   |
122 | |         None
123 | |     })
    | |_____- captured by this `FnMut` closure

My vague understanding is that I pass a mutable reference to from_eq which is also mutable outside of its scope, so now I would have two owners which is not ok.

I don't know how if my understanding is correct though or how to handle it even. I want to mutate chars in both functions, I am sure once from_eq has run, it's not mutating chars anymore—but how do I comply with the borrow checker here?

borrow-checker rust
2 Answers

Suppose from_eq() returned Some(...). Next iteration of the returned iterator, std::iter::from_fn() will call the closure again and it in its turn will call chars.next(). But chars was moved into from_eq() in the previous iteration!

This is what the compiler says: from_fn() takes a FnMut, and you cannot move a captured variable - chars - in a FnMut closure, because it can be called again and the variable will now be in invalid state.

The solution is to borrow chars in from_eq() instead of moving it:

fn from_eq<'a>(input: &'a str, start: usize, chars: &mut CharIndices) -> Option<Token<'a>> { ... }

'=' => from_eq(input, start, &mut chars),
user_78843050

You're not actually passing a mutable reference to from_eq. That's precisely the problem. You're moving chars into from_eq as a mutable argument. Once you've moved a value in Rust, you can't use it anymore, this is why you're getting the error.

But to use an iterator you only need a mutable reference to it, let's change from_eq's signature to reflect that:

fn from_eq<'a>(input: &'a str, start: usize, chars: &mut CharIndices) -> Option<Token<'a>> {
    // -- snip --
}

And in the calling code, we can use the by_ref() method provided by Iterator to pass chars in:

return match c {
    '(' => Some(Token::Lparen(&input[start..end])),
    ')' => Some(Token::Rparen(&input[start..end])),
    '[' => Some(Token::Lbrack(&input[start..end])),

    '='  => from_eq(input, start, chars.by_ref()),
    // --------------------------------^^^^^^^^^
    // pass in `chars` by mutable reference instead of by value

    // -- snip --

    _ => Some(Token::Unknown(&input[start..end])),
}
user_114231040
Related