Bad JWT signature trying to use the Google Drive API

I've seen articles about manipulating files with Google Drive using a personal OAuth2 token in PowerShell, but not with domain-wide privileges.

From what I understand, the workflow goes like this:

1. Create the header, base64 encode
2. Create the claim, base64 encode
3. Combine the header and claim, then sign it with RSA256 and the private key provided by Google

I'm having issues with step 3. Please see the below code:

$firstdate = Get-Date -Year 1970 -Month 1 -Day 1 -Hour 0 -Minute 0 -Second 0 -Millisecond 0
$endtime = (New-TimeSpan -Start $firstdate -End (Get-Date).AddMinutes(30).ToUniversalTime()).TotalSeconds
$issuetime = (New-TimeSpan -Start $firstdate -End (Get-Date).ToUniversalTime()).TotalSeconds
$jsonfile = Get-Content 'secret_file.json' | ConvertFrom-Json

$headerhash = [ordered]@{
    'alg' = 'RS256'
    'typ' = 'JWT'
}
$headerjson = $headerhash | ConvertTo-Json -Compress
$encodedheader = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($headerjson))

$claimhash = [ordered]@{
    'iss' = $jsonfile.client_email
    'scope' = 'https://www.googleapis.com/auth/drive'
    'aud' = 'https://www.googleapis.com/oauth2/v4/token'
    'exp' = [math]::Round($endtime,0)
    'iat' = [math]::Round($issuetime,0)
}
$claimjson = $claimhash | ConvertTo-Json -Compress
$encodedclaim = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($claimjson))

#$sha = [System.Security.Cryptography.RSACryptoServiceProvider]::Create()
#$sha.Key = [System.Text.Encoding]::UTF8.Getbytes($jsonfile.private_key)
#$sha.FromXmlString($sha.ToXmlString($jsonfile.private_key))
#$signature = $sha.ComputeHash([System.Text.Encoding]::UTF8.GetBytes("$encodedheader.$encodedclaim"))
#$signature = $sha.SignData([System.Text.Encoding]::UTF8.GetBytes("$encodedheader.$encodedclaim"),[System.Security.Cryptography.CryptoConfig]::MapNameToOID('RSASSA-PKCS1-V1_5-SIGN'))
#$encodedsignature = [System.Convert]::ToBase64String($signature)
#$formatter = [System.Security.Cryptography.RSAPKCS1SignatureFormatter]::new($sha)
#$formatter.SetKey($sha)
#$formatter.CreateSignature([System.Text.Encoding]::UTF8.GetBytes("$encodedheader.$encodedclaim"))

$jws = "$encodedheader.$encodedclaim"
$encodedjws = [System.Text.Encoding]::UTF8.GetBytes($jws)

$rsa = [System.Security.Cryptography.RSACryptoServiceProvider]::Create()
# This is the key -- need to convert PEM to proper format
$rsa.FromXmlString($rsa.ToXmlString($jsonfile.private_key))
$sha256OID = [System.Security.Cryptography.CryptoConfig]::MapNameToOID("SHA256")
$signature = $rsa.SignData($encodedjws,$sha256OID)
$encodedsignature = [System.Convert]::ToBase64String($signature)

$jwt = "$encodedheader.$encodedclaim.$encodedsignature"

$requestUri = 'https://www.googleapis.com/oauth2/v4/token'
$method = 'POST'
$body = [ordered]@{
    'grant_type' = 'urn:ietf:params:oauth:grant-type:jwt-bearer'
    'assertion' = $jwt
}
Invoke-webrequest -Uri $requestUri -Method $method -Body $body -ContentType application/x-www-form-urlencoded

The errors I'm getting are indicating a bad JWT signature. The secret key Google is providing is in PEM format, but I'm having issues translating it into something Powershell can understand.