Event when smart card of the X509Certificate2 object is removed

We have a feature where we use Windows' personal certificate store to authenticate using certificates after the user chooses a valid certificate. It simply signs a challenge with following:

public static byte[] SignDataSHA512RSA(X509Certificate2 certificate, byte[] data)
        {
            using (var rsa = certificate.PrivateKey as RSACryptoServiceProvider)
            {
                if (rsa == null)
                {
                    return null;
                }

                return rsa.SignData(data, CryptoConfig.MapNameToOID("SHA512"));
            }
        }

It works pretty well using the RSACryptoServiceProvider class. The feature allows both regular certificates and smart card certificates. If there is a smart card and it requires PIN, windows prompts a dialog for it and all.

Now, there is an extra security requirement in this feature where if a smartcard is used for this operation (which we can find out by rsa.CspKeyContainerInfo.Removable, and HardwareDevice if you want to enforce hardware), we want to make sure that it's accessible at all times during the session. In other words, we need an event when the smart card is removed so we can log out automatically.

One crude way would be having a timer job which checks certificate.PrivateKey is accessible every minute or so, but that doesn't throw before prompting user to insert the smart card and user presses Cancel.

The feature supports Windows 7 as well, so using the UWP libraries is not an option. Any way to accomplish this?