How to match all whitespace?

user_9080436 0
Viewed 695

Context: Rust has the match construct, which is really useful to make a (possibly) exhaustive list of cases and their corresponding results. The problem is: how do I create a case which encompasses a subset of many cases?

Regarding my specific problem, I'm making a lexer which reads a string character-by-character and spits out tokens. Its main function looks like this:

(...)
    fn new(input: &str) -> Lexer {
        let mut characters = input.chars();
        for c in characters {
            let mut token: Option<Token> = match c {
                '+' => Some(Token::Add),
                '-' => Some(Token::Minus),
                '*' => Some(Token::Mul),
                '/' => Some(Token::Div),
                'e' => Some(Token::EulersNum),
                'π' => Some(Token::Pi),
                '(' => Some(Token::LeftParen),
                ')' => Some(Token::RightParen),
                ' ' | '\t' | '\n' => continue, //Whitespace
                _ => None
            };

            if token == None {
                continue;
            }
        }

        todo!()
    }
(...)

Now, the most important part, for the purposes of this question, is the one commented with 'Whitespace'. The problem with my handling of the whitespaces is that it may not correspond to the actual implementation of whitespaces in a given string format. Sure, I could handle all of the different kinds of ascii whitespaces, but what about Unicode? Making an exhaustive list of whitespaces is something that is not only cumbersome, but also something that obfuscates the code. It should be left to the language, not to it's users.

Is it possible to just match it with a 'Whitespace' expression, such as:

(...)

Whitespace => continue,

(...)

And if so, how do I do it?

rust
2 Answers

You could use char::is_whitespace() in a match guard:

match c {
    '+' => Some(Token::Add),
    '-' => Some(Token::Minus),
    '*' => Some(Token::Mul),
    '/' => Some(Token::Div),
    c if c.is_whitespace() => Some(Token::Whitespace),
    _ => None,
};

Playground link

user_120152300

Sure, I could handle all of the different kinds of ascii whitespaces, but what about Unicode?

Use a library that provides a is_whitespace() function for many string formats or more complex matching capabilities, if you need it.

Otherwise, use functions like char::is_whitespace() if you just need to match against Unicode whitespace.

Making an exhaustive list of whitespaces is something that is not only cumbersome, but also something that obfuscates the code. It should be left to the language, not to it's users.

No, a match (and pattern matching etc.) are general tools. Rust is not a language specialized for language or string processing. Thus adding support for "whitespace matching" to pattern matching does not make sense.

Rust has basic ASCII, UTF-8, UTF-16 etc. support for practical reasons, but that's about it. Adding complex bits part of the standard library is debatable.

user_93053980
Related