AWS WAF not blocking traffic from blocked IPs

Here's the web ACL definition I'm using in json format. I'm blocking all traffic by default, then added 1 rule to allow traffic from a small range of IPs

{
  "Name": "app-*-WebACL",
  "Id": "fbb1f8f1-d78b-42de-8ebf-*",
  "ARN": "arn:aws:wafv2:us-east-1:*:global/webacl/app-*-WebACL/fbb1f8f1-d78b-42de-8ebf-*",
  "DefaultAction": {
    "Block": {}
  },
  "Description": "",
  "Rules": [
    {
      "Name": "OnlyAllowSomeIPs",
      "Priority": 0,
      "Statement": {
        "IPSetReferenceStatement": {
          "ARN": "arn:aws:wafv2:us-east-1:*:global/ipset/app-*-ipset/2d0e7033-9007-43ad-b755-*"
        }
      },
      "Action": {
        "Allow": {}
      },
      "VisibilityConfig": {
        "SampledRequestsEnabled": true,
        "CloudWatchMetricsEnabled": true,
        "MetricName": "OnlyAllowSomeIPs"
      }
    }
  ],
  "VisibilityConfig": {
    "SampledRequestsEnabled": true,
    "CloudWatchMetricsEnabled": false,
    "MetricName": "app-*-webACL"
  },
  "Capacity": 1,
  "ManagedByFirewallManager": false
}

When I hit the associated Cloudfront from the whitelisted IP, things work as expected. When I hit it from outside allowed range, I expect to be prevented access - I still am able to access content. I see a response header x-cache: Error from cloudfront but the content itself is being served. How do I make the block work?