Boost.Variant Vs Virtual Interface Performance

I'm trying to measure a performance difference between using Boost.Variant and using virtual interfaces. For example, suppose I want to increment different types of numbers uniformly, using Boost.Variant I would use a boost::variant over int and float and a static visitor which increments each one of them. Using class interfaces I would use a pure virtual class number and number_int and number_float classes which derive from it and implement an "increment" method.

From my testing, using interfaces is far faster than using Boost.Variant. I ran the code at the bottom and received these results:
Virtual: 00:00:00.001028
Variant: 00:00:00.012081

Why do you suppose this difference is? I thought Boost.Variant would be a lot faster.

** Note: Usually Boost.Variant uses heap allocations to guarantee that the variant would always be non-empty. But I read on the Boost.Variant documentation that if boost::has_nothrow_copy is true then it doesn't use heap allocations which should make things significantly faster. For int and float boost::has_nothrow_copy is true.

Here is my code for measuring the two approaches against each other.

#include <iostream>

#include <boost/variant/variant.hpp>
#include <boost/variant/static_visitor.hpp>
#include <boost/variant/apply_visitor.hpp>

#include <boost/date_time/posix_time/ptime.hpp>
#include <boost/date_time/posix_time/posix_time_types.hpp>
#include <boost/date_time/posix_time/posix_time_io.hpp>

#include <boost/format.hpp>

const int iterations_count = 100000;

// a visitor that increments a variant by N
template <int N>
struct add : boost::static_visitor<> {
    template <typename T>    
    void operator() (T& t) const {
        t += N;
    }
};

// a number interface
struct number {        
    virtual void increment() = 0;
};

// number interface implementation for all types
template <typename T>
struct number_ : number {
    number_(T t = 0) : t(t) {}
    virtual void increment() {
        t += 1;
    }
    T t;
};

void use_virtual() {
    number_<int> num_int;
    number* num = &num_int;

    for (int i = 0; i < iterations_count; i++) {
        num->increment();
    }
}

void use_variant() {
    typedef boost::variant<int, float, double> number;
    number num = 0;

    for (int i = 0; i < iterations_count; i++) {
        boost::apply_visitor(add<1>(), num);
    }
}

int main() {
    using namespace boost::posix_time;

    ptime start, end;
    time_duration d1, d2;

    // virtual
    start = microsec_clock::universal_time();
    use_virtual();
    end = microsec_clock::universal_time();

    // store result
    d1 = end - start;

    // variant
    start = microsec_clock::universal_time();
    use_variant();
    end = microsec_clock::universal_time();

    // store result
    d2 = end - start;

    // output
    std::cout << 
        boost::format(
            "Virtual: %1%\n"
            "Variant: %2%\n"
        ) % d1 % d2;
}