C# RSA Decryption Issue with Smart Card - Padding not removed from the result

I'm using the RSACryptoServiceProvider to encrypt and decrypt simple strings and while it works with normal certificates, I'm facing troubles when it comes with smart cards.

Here is the code that I'm using:

Create the RSACryptoServiceProvider:

        public static RSACryptoServiceProvider GetRSACSPSmartCard()
        {
            CspParameters csp = new CspParameters(1, "Microsoft Base Smart Card Crypto Provider", "XML_ENC_RSA_KEY");
            csp.Flags = CspProviderFlags.UseDefaultKeyContainer;
            csp.KeyPassword = GetSmartCardPin();
            RSACryptoServiceProvider rsa = new RSACryptoServiceProvider(csp);
            return rsa;
        }

Testmethod for encryption and decryption:

        private static void TestEnAndDecryption(RSACryptoServiceProvider rsa)
        {
            byte[] toEncryptData;
            byte[] encryptedRSA;
            
            Console.WriteLine(rsa.ToXmlString(false));
            Console.WriteLine();
            Console.WriteLine(rsa.CspKeyContainerInfo);
            toEncryptData = Encoding.ASCII.GetBytes("hello world");
            encryptedRSA = rsa.Encrypt(toEncryptData, RSAEncryptionPadding.Pkcs1);
         
            string EncryptedResult = Encoding.ASCII.GetString(encryptedRSA);
            Console.WriteLine(EncryptedResult);
            Console.WriteLine();

            byte[] decryptedRSA = rsa.Decrypt(encryptedRSA, RSAEncryptionPadding.Pkcs1);
            string originalResult = Encoding.ASCII.GetString(decryptedRSA);
            Console.WriteLine(originalResult);
        }

It does decrypt successfully but the in the result the padding is not removed: Output of the encryption/decryption

What's also interesting is no matter what I encrypt, the decrypted byte[] length is always 256. The same code works with a normal RSA key which is not from a smart card.

Does anyone of you have ever had a similar situation?

I'm grateful for any tips or support - Thanks.