AWS Cloudfront - S3 with CloudFront Origin Access and Referer bucket policy

I have this settings on my s3 bucket policy, where I allow only my cloudfront and referer to access the files on the bucket. I also used Signed URL.

{
    "Version": "2008-10-17",
    "Id": "PolicyForCloudFrontContent",
    "Statement": [
        {
            "Sid": "1",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity ABCD1234"
                ]
            },
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::my-files/*",
            "Condition":{
                "StringLike":{"aws:Referer":[
                    "https://example.com/*"
                ]}
            }
        }
    ]
}

And on the cloudfront behavior, I added the Referer here

Then, invalidate the cache.

Before setting the referer on s3 bucket policy, I can still access my files with signed url. But after setting up the aws:Referer Condition and invalidate the cache, I cannot access my files. It shows 403 Forbidden status.

What settings did I missed? Or is it possible to use Cloudfront OAI with Referer?