Override RSA private key operation for Smart Card?

I am implementing smart card middleware and I need to add RSA-PSS signature scheme to it. I wish I could override the code in Crypto++ which is responsible for private key operation and get everything else for free (PSS and PKCS padding). I figured this code in rsa.cpp is what I want to rewrite differently for smart cards:

Integer InvertibleRSAFunction::CalculateInverse(RandomNumberGenerator &rng, const Integer &x) const
{
    DoQuickSanityCheck();
    ModularArithmetic modn(m_n);
    Integer r, rInv;
    do {    // do this in a loop for people using small numbers for testing
        r.Randomize(rng, Integer::One(), m_n - Integer::One());
        rInv = modn.MultiplicativeInverse(r);
    } while (rInv.IsZero());
...

First I thought this can be done by subclassing RSA::PrivateKey

class MyPrivKey : public RSA::PrivateKey {
public:
    template<typename... Args>
    MyPrivKey(Args&&... args) : RSA::PrivateKey(std::forward<Args>(args)...) {}

    Integer CalculateInverse(RandomNumberGenerator &rng, const Integer &x) const override {
      return ... // do some smart card magic
    }
};

And then passing this key to the signer object

MyPrivateKey privateKey(params);

// Signer object
RSASS<PSS, SHA1>::Signer signer(privateKey);

But soon I found out that all components of privateKey (modulus, private and public exponent, etc.) are just copied to signer internals. So overriding RSA::PrivateKey::CalculateInverse doesn't really help.

Then I got lost in the templates and cried. I guess there's 0.0001% chance someone can help me with this. I'll try my luck.

Full code of my example https://pastebin.com/Nwk4jX0j