Change lambda from supported runtime to docker image

I have a lambda which was created and deployed with CloudFormation. The lambda, as of now, uses the supported .NET Core 3.1 runtime but I want to use the "unsupported" .Net Core 5 so I have created a Docker image which the lambda can run.

The problem I have is that CloudFormation can't seem to change the lambda from running a supported runtime to running a Docker image. I get Please don't provide ImageUri when updating a function with packageType Zip. when updating the CloudFormation stack with the updated template.

This is how my CloudFormation template looks like:

MyFunction:
    Type: "AWS::Serverless::Function"
    Properties:
        PackageType: Image
        ImageUri: <uri-to-ecr-docker-image>
        ...
    Metadata:
        Dockerfile: Dockerfile
        DockerContext: .
        DockerTag: v1

Any ideas on how to solve this?