Before I begin, let me note that I think there have been many related questions and answers, all of which proved useful to me. However, I couldn't find anyone who thoroughly described a method to do everything I wanted, so I thought I would document the problem and my solution and ask if there were better approaches.
Let's suppose that I have some slow--but definitely correct--code to perform a certain task that I keep in my project to test a faster implementation. For concreteness, define:
pub fn fast(x: T) -> U {
// does stuff and eventually returns a value `out`
// ...
debug_assert_eq!(out, slow(x))
out
}
#[cfg(any(test, debug_assertions))]
pub fn slow(x: T) -> U { ... }
This is all fine and good. However, now suppose that I would like to add some benchmarks to demonstrate how good my fast implementation is...
Attempt 1: Criterion
I think that a standard way to set up benchmarking is to put a benches/ directory in the project, add a [[bench]] to Cargo.toml with the harness disabled, and use the criterion crate. However, if I understand correctly, if we then run cargo bench, the benchmark will have to take the position of a user that cannot access crate features defined only during testing. Thus, slow will not be resolved and the command will fail.
A Quick Aside: Another thing that derailed me for a while is that I kept wanting to use bench as a cfg flag but couldn't find anything about this. As it turns out, I think that test also covers the benching case. (I think all the seasoned rustaceans will be laughing at me, but this seems like a useful thing to note for anyone in a similar situation).
Attempt 2: The Nightly Test Crate
Since the previous method didn't seem fruitful, another popular option seemed to be to use the unstable test crate. This results in a project structure that looks like:
Cargo.toml
src/
lib.rs
bench.rs
Our original file is then revised to be:
// Include the unstable feature
#![feature(test)]
pub fn fast(x: T) -> U { ... }
#[cfg(any(test, debug_assertions))]
pub fn slow(x: T) -> U { ... }
#[cfg(test)]
mod bench;
And then bench.rs should look something like:
extern crate test;
use test::Bencher;
#[bench]
fn bench_it(b: &mut Bencher) {
b.iter(|| {}) // gotta go fast
}
This seemed to do everything I wanted upon running cargo +nightly bench. However, it is also super desirable for the project to be compilable outside of testing without the use of nightly or extra feature flags. That is, I still want to be able to run cargo build and cargo test and not get yelled at for requesting unstable features on a stable channel.
Attempt 2.5: Enter Build Scripts
(Once again, each of the parts is well-documented in other questions, I'm just collecting everything here for fun). Using a bunch of other posts, I learned that we can check for nightly and conditionally enable features by way of a build script. Our project now looks like this:
Cargo.toml
build.rs
src/
lib.rs
bench.rs
And we need to add rustc_version to our [build-dependencies] in Cargo.toml. We then add the following build script:
use rustc_version::{version_meta, Channel};
fn main() {
// Set feature flags based on the detected compiler version
match version_meta().unwrap().channel {
Channel::Stable => {
println!("cargo:rustc-cfg=RUSTC_IS_STABLE");
}
Channel::Beta => {
println!("cargo:rustc-cfg=RUSTC_IS_BETA");
}
Channel::Nightly => {
println!("cargo:rustc-cfg=RUSTC_IS_NIGHTLY");
}
Channel::Dev => {
println!("cargo:rustc-cfg=RUSTC_IS_DEV");
}
}
}
Finally, if we update lib.rs to be the following:
// Include the unstable feature
#![cfg_attr(RUSTC_IS_NIGHTLY, feature(test))] // <-- Note the change here!
pub fn fast(x: T) -> U { ... }
#[cfg(any(test, debug_assertions))]
pub fn slow(x: T) -> U { ... }
#[cfg(all(RUSTC_IS_NIGHTLY, test))] // <-- Note the change here!
mod bench;
I think we get everything we want.
So... thanks for joining me on this adventure. Would appreciate commentary on whether or not this was the right approach. Also, you might ask "why keep the benchmark around once we know it's slower?" I suppose this might be fair, but perhaps the test could be changed or I'd like to prove the new implementation is faster to a third party that won't just trust me.