AWS S3 File Sync fails from Github Action but works fine locally

I'm trying to use this Github Action (https://github.com/jakejarvis/s3-sync-action) to run the aws cli s3 sync command to push files for a static website to an S3 bucket.

I setup the bucket policy to allow the required operations used by the sync command:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadAccess",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::my-bucket/*"
        },
        {
            "Sid": "GithubDeploymentA",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::163936899089:user/my-deploy-user"
            },
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::my-bucket"
        },
        {
            "Sid": "GithubDeploymentB",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::163936899089:user/my-deploy-user"
            },
            "Action": [
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::my-bucket*"
        }
    ]
}

Now the sync command works when I locally run it using the "my-deploy-user" credentials. It even works if I run the docker image for the Github Action locally so I've verified that the script works, the user is setup correctly, and the bucket policy is setup correctly. However when running from the Github Action the script fails with the s3 sync command with Access Denied on the PutObject operations:

upload failed: build/favicon.ico to s3://my-bucket/favicon.ico An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

This is my first time really using AWS so maybe I'm missing something obvious but couldn't find anything googling so far. Since the Github Action is using the same credentials for the script as when I run it locally and I haven't setup any IP related conditions for permissions I'm not sure what to look at next.

Any ideas?