How can I explicitly Deny lambda:InvokeFunction except for trusted resources with IAM?

I have been trying to figure out how to create a highly restrictive IAM policy but seem to be running into issues.

We have a lambda that returns sensitive information. A few other lambdas in our system (currently only 3) will need the ability to invoke this handler directly with the lambda:InvokeFunction permission but we want to make it very explicit which functions have access.

Our goal is to have an explicit Deny IAM policy that whitelists the functions that should be granted access. This way, we can centrally manage the whitelist rather than relying on devs to create Allow policies for themselves.

I've tried the following however it seems to block access to everything, including the whitelisted ARN.

[
  {
    Effect: 'Allow',
    Action: ['lambda:InvokeFunction'],
    Resource: 'arn:aws:lambda:us-east-1:123456789:function:protected-function',
  },
  {
    Effect: 'Deny',
    Action: ['lambda:InvokeFunction'],
    // the resource we are protecting
    Resource: 'arn:aws:lambda:us-east-1:123456789:function:protected-function',
    Condition: {
      ArnNotLike: {
        // the lambda that should have access to Invoke
        'AWS:SourceArn': ['arn:aws:lambda:us-east-1:123456789:function:access-protected-data'],
      },
    },
  }
]

What would be the best way to secure this function using IAM to ensure that we can have central management of permissions while still allowing our devs to deploy via a shared CI/CD IAM user that is responsible for provisioning the stack. Open to any ideas that help us secure the function - including protection against any possible internal bad actors/errors.