How to deploy S3 Bucket with AWS CDK that is only available to specific IP addresses (NotIpAddress condition)?

cdk deploy command fails whenever I add the following policy:

s3BucketOfFrontend.addToResourcePolicy(new iam.PolicyStatement({
    effect: Effect.DENY,
    actions: ['s3:*'],
    resources: [s3BucketOfFrontend.arnForObjects('*')],
    principals: [new iam.AnyPrincipal()],
    conditions: {
        'NotIpAddress': {
            'aws:SourceIp': definitions.permittedProxyIPs // list of IP strings
        }
    }
}))

With this error:

Custom::CDKBucketDeployment | my-cdk-ts-deployment-bucket/CustomResource/Default (mycdktsdeploymentbucketCustomResource1FF9A593) Failed to create resource. Command '['python3', '/var/task/aws', 's3', 'sync', '--delete', '/tmp/tmpvs26w_jk/contents', 's3://my-frontend-stack-mycdktsbucket46f56458-1dxm7rpoe13nf/']' returned non-zero exit status 1

Elaboration:

My S3 bucket must only be available to specific IP addresses. I'm trying to deploy it using CDK:

cdk synth
cdk bootstrap --public-access-block-configuration false # otherwise I get CREATE_FAILED | StagingBucket API: s3:PutPublicAccessBlock Access Denied
cdk deploy

I'm trying to get to this policy (on the S3 Bucket):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": [
                "arn:aws:s3:::my-frontend-stack-mycdktsbucket46f56458-4j64761048fr/*"
            ]
        },
        {
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": [
                "arn:aws:s3:::my-frontend-stack-mycdktsbucket46f56458-4j64761048fr/*"
            ],
            "Condition": {
                "NotIpAddress": {
                    "aws:SourceIp": [
                        "123.123.123.123/27",
                        "123.123.124.123/27"
                    ]
                }
            }
        }
    ]
}

Here's the full error I see when running the cdk deploy command:

 6/8 | 12:20:56 | CREATE_FAILED        | Custom::CDKBucketDeployment | my-cdk-ts-deployment-bucket/CustomResource/Default (mycdktsdeploymentbucketCustomResource1FF9A593) Failed to create resource. Command '['python3', '/var/task/aws', 's3', 'sync', '--delete', '/tmp/tmpvs26w_jk/contents', 's3://my-frontend-stack-mycdktsbucket46f56458-1dxm7rpoe13nf/']' returned non-zero exit status 1.
        new CustomResource (C:\my\prj\tib\cdk\node_modules\@aws-cdk\aws-s3-deployment\node_modules\@aws-cdk\core\lib\custom-resource.ts:115:21)
        \_ new BucketDeployment (C:\my\prj\tib\cdk\node_modules\@aws-cdk\aws-s3-deployment\lib\bucket-deployment.ts:201:5)
        \_ new CdkStack (C:\my\prj\tib\cdk\lib\cdk-stack.ts:17:9)
        \_ Object.<anonymous> (C:\my\prj\tib\cdk\bin\cdk.ts:12:1)
        \_ Module._compile (internal/modules/cjs/loader.js:1251:30)
        \_ Module.m._compile (C:\my\prj\tib\cdk\node_modules\ts-node\src\index.ts:858:23)
        \_ Module._extensions..js (internal/modules/cjs/loader.js:1272:10)
        \_ Object.require.extensions.<computed> [as .ts] (C:\my\prj\tib\cdk\node_modules\ts-node\src\index.ts:861:12)
        \_ Module.load (internal/modules/cjs/loader.js:1100:32)
        \_ Function.Module._load (internal/modules/cjs/loader.js:962:14)
        \_ Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:72:12)
        \_ main (C:\my\prj\tib\cdk\node_modules\ts-node\src\bin.ts:227:14)
        \_ Object.<anonymous> (C:\my\prj\tib\cdk\node_modules\ts-node\src\bin.ts:513:3)
        \_ Module._compile (internal/modules/cjs/loader.js:1251:30)
        \_ Object.Module._extensions..js (internal/modules/cjs/loader.js:1272:10)
        \_ Module.load (internal/modules/cjs/loader.js:1100:32)
        \_ Function.Module._load (internal/modules/cjs/loader.js:962:14)
        \_ Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:72:12)
        \_ C:\Program Files\nodejs\node_modules\npm\node_modules\libnpx\index.js:268:14