Save/load compiled Regex in Rust?

user_1103561 0
Viewed 348

I'm working on some application where performance is critical and i'm looking for opportunities to improve it. The app uses regular expressions extensively (regex crate), eg.:

use regex::Regex;
...
let Regex = Regex::new(r###"[a-z0-9%]{3,}"###).unwrap();

Is there any opportunity to save a precompiled Regex instance to a buffer (think Vec<u8>) and load later pre-compiled to avoid multiple compilations (can't see anything similar in the docs)?

PS. I already do it lazily with:

    lazy_static! {
        static ref REGEX2: Regex = Regex::new(r###"[a-z0-9%]{3,}"###).unwrap();
    }

trying to save some CPU cycles if having multiple invocations (but that's another story).

PS. Just out of curiosity i've benchmarked the code:

fn bench_regex_candidates(b: &mut Bencher) {
    b.iter(|| {
        Regex::new(r###"[a-z0-9%]{3,}"###).unwrap();
    });
}

fn bench_regex_content(b: &mut Bencher) {
    b.iter(|| {
        Regex::new(r###"^([^*|@"!]*?)#([@?$])?#(.+)$"###).unwrap()
    });
}

and the compilation is fast (relatively, for my needs):

regex/candidates        time:   [22.021 us 22.684 us 23.689 us]                              
Found 11 outliers among 100 measurements (11.00%)
  2 (2.00%) high mild
  9 (9.00%) high severe
regex/content           time:   [36.542 us 36.687 us 36.845 us]                           
Found 8 outliers among 100 measurements (8.00%)
  3 (3.00%) high mild
  5 (5.00%) high severe
rust deserialization serialization regex
1 Answers

The short answer is that no, you cannot. Something like this has to be provided by the regex crate itself. There is an open issue tracking it: https://github.com/rust-lang/regex/issues/258

It's unlikely a feature like this will be supported any time soon. Generally speaking, there are two ways that the regex crate could go about this. The first would be to simply serialize its internal data structures using Serde. The second would be to make the internal implementation congruent to a bespoke serialized structure.

In the first case, the implementation effort of the naive approach probably isn't too bad, and could likely be accomplished by sprinkling a few derive(serde::Serialize, serde::Deserialize) in a few places. There are two main problems with this approach. Firstly, it requires an explicit serialization step that will have costs of its own, and it's not clear whether it would be meaningfully faster than regex compilation itself in enough cases. Secondly, this would expose internal data structures as a public API guarantee. So in actuality, implementing this would probably require choosing a format that we could commit to. This would in turn likely require yet another translation step before serialization and after deserialization, increasing costs. So overall, this approach requires a fair amount of effort and it's not clear whether it's worth it.

The second approach would make serialization and deserialization effectively free, because the internal representation would be the serialized form. This suffers from the same "public API" format guarantee of the first problem, so it would have to be done carefully in order to manage compatibility guarantees. The other main problem with this approach is that it infects the entirety of the internal implementation. Effectively, everything must be written with this sort of constraint in mind. Internal pointers, for example, could no longer be used. So there is a big trade off in code complexity required to do this. And it's also a lot of work.

Others have mentioned benchmarking your code. You should do this, and this question would be a much better question if you could carefully justify why you need to eliminate the compilation cost. Just because you're writing a high performance application and you're using regexes compiled with pattern strings at runtime doesn't mean regex compilation is itself a bottleneck. It absolutely could be, but it's also pretty likely that it isn't. Therefore, it's really important to check this assumption. Ideally, you would create a minimal benchmark program that reproduces the same usage patterns of your application, and then measure which aspects of your program use the majority of the time.

With all that said, the regex-automata crate does support (de)serializing regexes, and it does so using the second approach I outlined above. However, regex-automata is not a general purpose regex engine and there are severe differences between it and the regex crate. For example, it would not be appropriate to use regex-automata to compile regexes with pattern strings derived from user input, since compilation could take exponential time in the size of the pattern.

user_6192160
Related