Speedup an if-else ladder C++

A few optimizations that come to mind:

1. vector.push_back() or emplace_back(), even with reserve(), are poison for performance because no compiler is able to vectorize the code. We can work with plain C pointers instead or just preallocate.

2. Generating the random engine and distribution in the last case may have significant cost if this code is called repeatedly. We can hoist this out of the code. Note that this will also avoid issues with the repeated initialization for which you use a low-resolution time function.

3. This may be unnecessary but rewriting the code a bit may allow more compiler optimizations, especially by turning things into conditional move-instructions and reducing the number of branches.

/* TODO: We have better ways of initializing generators but that is
 * unrelated to its performance
 * I'm lazy and turn this into a static variable. Better use a
 * different pattern (like up in the stack somewhere)
 * but you get the idea
 */
static default_random_engine generator(time(0));
static normal_distribution<float> dist(0,1);

std::size_t output_pos = filtered_output.size();
filtered_output.resize(output_pos + WINDOW_SIZE);
flag_output.resize(output_pos + WINDOW_SIZE);

switch(RPL_OPTION) {
case 0:
    for(int k = 0; k < WINDOW_SIZE; k++) {
        auto ansk = ans[k];
        int flag = (ansk >= upper_th) | (ansk < lower_th);
        flag_count += flag;
        filtered_output[output_pos + k] = ansk;
        flag_output[output_pos + k] = flag;
    }
    break;
case 1:
    for(int k = 0; k < WINDOW_SIZE; k++) {
        auto ansk = ans[k];
        int flag = (ansk >= upper_th) | (ansk < lower_th);
        flag_count += flag;
        // written carefully to help compiler turning this into a CMOV
        auto filtered = flag ? RPL_CONST : ansk;
        filtered_output[output_pos + k] = filtered;
        flag_output[output_pos + k] = flag;
    }
    break;
case 2:
    for(int k = 0; k < WINDOW_SIZE; k++) {
        auto ansk = ans[k];
        int flag = (ansk >= upper_th) | (ansk < lower_th);
        flag_count += flag;
        auto filtered = ansk < lower_th ? lower_th : ansk;
        filtered = ansk >= upper_th ? upper_th : filtered;
        filtered_output[output_pos + k] = filtered;
        flag_output[output_pos + k] = flag;
    }
    break;
case 3:
    for(int k = 0; k < WINDOW_SIZE; k++) {
        // optimized under the assumption that flag is usually 1
        auto ansk = ans[k];
        auto random = dist(generator) * sigma;
        int flag = (ansk >= upper_th) | (ansk < lower_th);
        auto filtered = flag ? random : ansk;
        filtered_output[output_pos + k] = filtered;
        flag_output[output_pos + k] = flag;
    }
    break;
}

Analyzing compiler output

I checked the resulting code with Godbolt. Cases 0-2 do vectorize. However, a lot hinges on good alias detection. So this needs to be analyzed in the context of the full function containing this code. Particular pain points are

• Potential alias between ans and filtered_output. That is hard to avoid but I think compilers should be able to create code that check against this
• Potential alias between the thresholds + RPL_CONST and the filtered_output. When in doubt, copy the inputs into a local variable (which the compiler can prove to be alias free). Just marking them const may not be enough
• Potential alias between flag_count and flag_output, depending on the types. Again, better use a local variable for the count, then copy it to its output, if required

As for case 3, computing a random sample is expensive enough that my optimization may degrade performance if the inputs are usually within limits. That needs benchmarking. The longer I think about it, losing a few clock cycles on a mis-predict is probably much less time than computing a sample without using it.

Removing redundant code

The resulting code is highly redundant. We could move the switch-case into the loop but that messes with the vectorization. Instead, we can use a template function pattern.

class Filter
{
    int WINDOW_SIZE;
    float upper_th, lower_th, sigma, RPL_CONST;
    std::default_random_engine generator;
    std::normal_distribution<float> dist;

    template<class FilterOp>
    int apply(std::vector<float>& filtered_output,
              std::vector<int>& flag_output,
              const std::vector<float>& ans, FilterOp filter)
    {
        // move stuff into local variables to help with alias detection
        const int WINDOW_SIZE = this->WINDOW_SIZE;
        const float upper_th = this->upper_th, lower_th = this->lower_th;
        const std::size_t output_pos = filtered_output.size() - WINDOW_SIZE;
        int flag_count = 0;
        for(int k = 0; k < WINDOW_SIZE; k++) {
            auto ansk = ans[k];
            int flag = (ansk >= upper_th) | (ansk < lower_th);
            flag_count += flag;
            filtered_output[output_pos + k] = filter(ansk, flag);
            flag_output[output_pos + k] = flag;
        }
        return flag_count;
    }
public:
    int operator()(int RPL_OPTION,
              std::vector<float>& filtered_output,
              std::vector<int>& flag_output,
              const std::vector<float>& ans)
    {
        std::size_t output_pos = filtered_output.size();
        filtered_output.resize(output_pos + WINDOW_SIZE);
        flag_output.resize(output_pos + WINDOW_SIZE);
        switch(RPL_OPTION) {
        case 0:
            return apply(filtered_output, flag_output, ans,
                [](float ansk, int flag) noexcept -> float {
                    return ansk;
            });
        case 1:
            return apply(filtered_output, flag_output, ans,
                [RPL_CONST=this->RPL_CONST](float ansk, int flag) noexcept -> float {
                    return flag ? RPL_CONST : ansk;
            });
        case 2:
            return apply(filtered_output, flag_output, ans,
                [lower_th=this->lower_th, upper_th=this->upper_th](
                      float ansk, int flag) noexcept -> float {
                    auto filtered = ansk < lower_th ? lower_th : ansk;
                    return ansk >= upper_th ? upper_th : filtered;
            });
         case 3:
            return apply(filtered_output, flag_output, ans,
                [this](float ansk, int flag) noexcept -> float {
                    return flag ? dist(generator)*sigma : ansk;
            });
         default: return 0;
       }
    }
};

I am nearly 98% sure that the if-else ladder is not the problem.

The std::vectors (or whatever container you use) push_back function with tons of reallocations and data copying is for me the main candidate for optimization.

Please use the reserve function to allocate the needed memory beforehand.

Then move out all invariant stuff, like

default_random_engine generator(time(0));
normal_distribution<float> dist(0,1);

But without more example code, it is hard to judge.

A profiler will give you better results. Timer functions will not help a lot here.

I have a piece of code that demands execution speed over anything else

That suggests a non-obvious approach. The code pattern looks familiar enough from a signal processing viewpoint, so WINDOW_SIZE is likely non-trivial. In that case, using AVX2 with packed comparisons makes sense.

In short, you pack a whole AVX2 register full of inputs, use two AVX2 registers to store copies of the lower and upper threshold, and issue the two comparisons. This gives you two outputs, where each value is either 0 or ~0.

Hence, your flag count be determined by or-ing the two registers. It's tempting to count the flags already, but this is considered a slow "horizontal add". Better to keep track of this in another AVX register, and do one horizontal add at the end.

The updates to filtered_output depend on the case, but for 1 and 2 you can use AVX for this as well. Choosing between two values based on the bits in a third register can be done with mm256_blendv_epi8. You can safely ignore the 8 there, that's the minimum resolution (one byte). If you're doing 32 bits comparisons, your result register will also contain 32 bits outcomes, so mm256_blendv_epi8 will work with 4*8 bits resolution.

If you have case 0: of course should just be a straight copy to filtered_output, outside the if statements.

The first thing to notice is that you push_back on vectors. The code shows no call to reserve so this will resize the vector as it grows over and over. That's probably more expensive than anything else in the loop.

The next things concerns the "if-else-ladder":

Have you actually profiled if the ladder is a problem at all? Branches are only expensive when they mispredict. Maybe the branch predictor works juse fine on your input? Assuming this switch statement is executed many times one way to help it out would be to sort the input. Then the if-else-ladder wouldn't jump randomly every time but repeat the same way many times before switching to a new case. But that only helps if the loop runs many times or the cost of sorting will negate any improvement.

And if the switch is repeated many times you can split the input into 3 groups once, for the 3 options in the ladder, and process each group without any if-else-ladder.

Looking closer at the code I see the first 2 cases in the if-else-ladder are identical (except case 2). So you can combine the tests making this a simple "if-else":

if ((ans[k] >= upper_th) || (ans[k] < lower_th))

Now due to lazy evaluation this will produce the same code as before. But you can do better:

if ((ans[k] >= upper_th) | (ans[k] < lower_th))

Now both parts get evaluated since there is no lazy evaluation of |. Except compilers are artifical stupids and might just do lazy evaluation anyway. At this point you are fighting the optimizer. Some compilers will optimize this into 2 branches, some leave it at one.

You can use something like the following trick there:

static auto fn[] = {
    [&]() { code for first choice; };
    [&]() { code for second choice; };
};
fn[((ans[k] >= upper_th) | (ans[k] < lower_th))]();

By turning the condition of your if into computing the index of an array the compiler optimization producing 2 branches is circumvented. Hopefully. At least till the next compiler update. :)

When fighting the optimizer you have to recheck your solution every time you update the compiler.

And for case 2 the difference in the code is only what value to push_back. That can be turned into a conditional move on most architectures instead of a branch if you use

(ans[k] >= upper_th) ? upper_th : lower_th;

for the push_back.

Firstly, sorting ans could be a good idea because of bench prediction. But it is up to mostly size of the ans . Secondly, if you are using c++20 you can take a look at [LIKELY] and [UNLIKELY] keywords. If you can select which statement is picking mostly or the opposite you can easily use them.