Validating AuthToken from sign in with apple JS on .net Backend

I'm trying to implement Sign in with apple on our Web App, It works on AppleTV and IOS but the same flow doesn't seem to work with the token from Apple JS and I get INVALID_CLIENT.

It seems it needs a few extra steps so I added the KID and such and now I'm stuck on INVALID_GRANT (which is better according to what I understand at least the client is Ok).

From the Js side I do the following : (The flow seems correct with appropriate response in AppleIDSignInOnSuccess)

AppleID.auth.init({  
                 clientId: 'com.#########.weblogin',//service Id created   
                  scope: 'name email',  
                  state: state,  
                  redirectURI: location.href,  
                  usePopup: true //or false defaults to false  
              }); 



document.addEventListener('AppleIDSignInOnSuccess', (data) => {  
            //handle successful response  
            var deviceKey = document.cookie.replace(/(?:(?:^|.*;\s*)deviceKey\s*\=\s*([^;]*).*$)|^.*$/, "$1");  
            requestLogin({  
                AppleToken: data.detail.authorization.id_token,  
                AppleAuthorizationCode: btoa(data.detail.authorization.code),//our api requires base64urlsafe strings to be symetric with native apps
                FirstName: data.detail.user?.firstName,  
                LastName: data.detail.user?.LastName,  
                DeviceKey:  deviceKey  
            });  
        });

To be noted though that unlike the documentation [state] isn't returned, it might not be necessary in case of popup but that could be a sign that something is wrong.

Now I handle the AuthorizationCode with my .NET backend.

Hclient.DefaultRequestHeaders.Add("User-Agent", "Microsoft ASP.NET Core OpenIdConnect handler");  

var datas = new Dictionary<string, string="">()  
                        {  
                            { "client_id" , request.DeviceKey.StartsWith("WEB_") ? "com.########.weblogin" : "com.########.app" }, //use the serviceId and not the main appId for web  
                            { "code" , request.AppleAuthorizationCode.FromBase64UrlSafe().FromUtf8Bytes() },  
                            { "grant_type" , "authorization_code" },  
                            { "redirect_uri", "https://login.#######.com/signin-apple" },  
                            {"client_secret",request.DeviceKey.StartsWith("WEB_") ? TokenGenerator2.CreateNewToken() : TokenGenerator.CreateNewToken() }//generate the client_secret differently for web  
                        };  
var formdata = new FormUrlEncodedContent(datas);  

using (HttpResponseMessage res = Hclient.PostAsync("https://appleid.apple.com/auth/token",formdata ).Result)

This is the call that respond "INVALID_GRANT". And this Is how I generate my jwt token client_secret.

public static class TokenGenerator2  
        {  
            public static string CreateNewToken()  
            {  
                const string iss = "7#######G"; // team ID  
                const string aud = "https://appleid.apple.com";  
                const string sub = "com.#######.weblogin"; // service Id  
                const string keyId =  "G######W";//key Idassociated with the p8 file  

                const string privateKey = "MIGT####...#####"; // contents of AuthKey_[keyId].p8 file  



                var d = DateTime.UtcNow.AddDays(-5);//I was worried the date time was the issue so I took a laaaaarge one ...  


                var cngKey = CngKey.Import(  
                  Convert.FromBase64String(privateKey),  
                  CngKeyBlobFormat.Pkcs8PrivateBlob);  


                var handler = new JwtSecurityTokenHandler();  


                var securityKey = new ECDsaSecurityKey(new ECDsaCng(cngKey) { KeySize = 256 , HashAlgorithm = CngAlgorithm.ECDsaP256});  


                securityKey.KeyId = keyId;  
                var signingCredentials = new SigningCredentials(securityKey, SecurityAlgorithms.EcdsaSha256);  


                return  handler.CreateEncodedJwt(iss, aud, new ClaimsIdentity(new List { new Claim("sub", sub) }),d, expires: d.AddMonths(3),d, signingCredentials: signingCredentials);  

            }  

        }  

For reference this is how I create it for the native apps (the flow that do work)

  public static class TokenGenerator  
        {  
            public static string CreateNewToken()  
            {  
                const string iss = "7#######G"; // your account's team ID found in the dev portal  
                const string aud = "https://appleid.apple.com";  
                const string sub = "com.######.app";   
                const string privateKey = "MIGTAg###...####"; // contents of .p8 file  


                var cngKey = CngKey.Import(  
                  Convert.FromBase64String(privateKey),  
                  CngKeyBlobFormat.Pkcs8PrivateBlob);  
                var d = DateTime.UtcNow.AddDays(-5);  


                var handler = new JwtSecurityTokenHandler();  
                var token = handler.CreateJwtSecurityToken(  
                    issuer: iss,  
                    audience: aud,  
                    subject: new ClaimsIdentity(new List { new Claim("sub", sub) }),  
                    expires: d.AddMonths(3), // expiry can be a maximum of 6 months  
                    issuedAt: d,  
                    notBefore: d,  
                    signingCredentials: new SigningCredentials(  
                      new ECDsaSecurityKey(new ECDsaCng(cngKey)), SecurityAlgorithms.EcdsaSha256));  


                return handler.WriteToken(token);  
            }  


        }