Inheritance and templates instanciations with pointers to simulate "virtual data"

I have a hierarchy of classes:

class Base
{
public:
    Base():a{5}{}
    virtual ~Base(){};
    int a;
};

class Derived : public Base
{
public:
    Derived():b{10}{}
    int b;
};

I then have a class template that operates on whatever type it is instanciated with:

template<typename T>
class DoStuff
{
public:
    DoStuff():val{}{}
    virtual ~DoStuff(){};
    virtual void printDoStuff() = 0;
    T getVal(){return val;};
private:
    T val;
};

class DoStuffWithInt : public DoStuff<int>
{
public:
    virtual void printDoStuff() override {cout << "val = " << getVal() << endl;}
};

class DoStuffWithBase : public DoStuff<Base>
{
public:
    virtual void printDoStuff() {cout << "a = " << getVal().a << endl;}
};

Now I would like to have a hierarchy of class like this:

class DoStuffWithBase : public DoStuff<Base>
{
public:
    virtual void printDoStuff() {printVal(); cout << "a = " << getVal().a << endl;}
};

// Wrong and will not compile, trying to make a point
class DoStuffWithDerived : public DoStuffWithBase<Derived>
{
public:
    void printDoStuff() override {DoStuffWithBase::printDoStuff(); cout << "b = " << getVal().b << endl;}
};

Basically I would like to have DoStuffWithBase that operates on a base be extended so that I can reuse its functions, but the extended class DoStuffWithDerived should operate on a Derived type.

I managed to get something working by templating DoStuffWithBase with a pointer to Base and extending it:

template <class T>
static void deleteIfPointer(const T& t)
{
    std::cout << "not pointer" << std::endl;
}

template <class T>
static void deleteIfPointer(T* t)
//                           ^
{
    std::cout << "is pointer" << std::endl;
    delete t;
}

template<typename T>
class DoStuff
{
public:
    DoStuff():val{}{}
    DoStuff(const T& value):val{value}{};
    virtual ~DoStuff(){deleteIfPointer(val);}
    virtual void printDoStuff() = 0;
    T getVal(){return val;};
private:
    T val;
};

class DoStuffWithBase : public DoStuff<Base*>
{
public:
    // New base
    DoStuffWithBase(): DoStuff(new Base()){}
    DoStuffWithBase(Base* b) : DoStuff(b){}
    virtual void printDoStuff() {printVal(); cout << "a = " << getVal()->a << endl;}
};

class DoStuffWithDerived : public DoStuffWithBase
{
public:
    // New derived
    DoStuffWithDerived(): DoStuffWithBase(new Derived()){}
    void printDoStuff() override {DoStuffWithBase::printDoStuff(); cout << "b = " << static_cast<Derived*>(getVal())->b << endl;}
};

It works but there are several things I don't like:

1. The code is a lot more complicated, when 99% of the time, I won't need to extend a DoStuffWithX class, I will just use DoStuffWithInt, DoStuffWithClass, DoStuffWithAnotherClass etc... Here I had to add several constructors, a special case destructor and so on.

2. I have to use pointers and manage them (static_cast when needed, deletion...), all in order to avoid slicing and get the right type. Also, DoStuff::val should theorically not be null, but with a pointer there is no way I can prevent that (or atleast I don't know one). Maybe using smart pointers would help a bit here ? I am not super familiar with them.

3. I have to manage cases where T is a pointer and when it is not. For example, the deleteIfPointer function above, but also switching between . and -> and probably more.

Is there any simpler way to achieve what I am trying to do ? A design pattern or something else ? Am I stuck with my solution and is it somewhat good ?

Edit: I tried to implement it with std::variant as in @Tiger4Hire's answer:

class Derived : public Base
{
public:
    Derived():b{10}{}
    int b;
};

class Derived2 : public Base
{
public:
    Derived2():c{12}{}
    int c;
};

using DerivedTypes = std::variant<Derived, Derived2>;

struct VariantVisitor
{
    void operator()(Derived& d)
    {
        d.b = 17;
    }

    void operator()(Derived2& d)
    {
        d.c = 17;
    }
};

class DoStuffWithVariant : public DoStuff<DerivedTypes>
{
public:
    void handleBasePart(Base& base)
    {
        cout << "a = " << base.a << endl;
        base.a = 10;
    }

    virtual void printDoStuff() override
    {
        auto unionVal_l = getVal();

        if (std::holds_alternative<Derived>(unionVal_l))
        {
            std::cout << "the variant holds a Derived!\n";
            auto& derived_l = std::get<0>(unionVal_l);
            cout << "b = " << derived_l.b << endl;
            handleBasePart(derived_l);
        }
        else if (std::holds_alternative<Derived2>(unionVal_l))
        {
            std::cout << "the variant holds a Derived2!\n";
            auto& derived2_l = std::get<1>(unionVal_l);
            cout << "c = " << derived2_l.c << endl;
            handleBasePart(derived2_l);
        }
        std::visit(VariantVisitor{}, unionVal_l);
    }
};

What I like about it:

1. I don't have to use pointers.
2. I feel the code is less tricky, easier to understand.

What I don't like about it:

1. The code is all in one place and it deals with all the possible Derived types (and even the Base type) at once whereas with inheritance, classes are more specialized, you can really look at a class and directly know what it does, what it overrides etc... On the other hand one could argue that it means the algorithm is in one place instead of dispatched all over the classes hierarchy.
2. You can't have an abstract base class as your interface.

All in all it is a really good alternative, but I am still wondering if there is a simpler way to implement dynamic polymorphism ? Do you necessarily have to resort to (base class) pointers with dynamic polymorphism ? Are std::variant the way to go now ?

Edit2: 2 other drawbacks with variants that I didn't notice at first:

1. All your derived class and your base class have to be defined in the same library. Clients can't easily add a new Derived class since it would mean modifying the variant and they might not have access to it. On the project I am working on, base classes are defined in one library, and are derived in other independant "sub" libraries. So if I try to use variant in my main library, it won't be able to access the Derived types in the sub libraries, which is a major issue.
2. If your base class implenting the variant (DoStuff here) has other members, when you call std::visit on the variant, you might have to also embark the needed other members of DoStuff. I think you should be able to use lambdas to capture them, but still, it's a lot less straightforward than using them directly as in the case of inheritance.