403 Forbidden. when serving multipage app via cloudfront from s3

In general everything works, I have a gatsby generated static website, s3 bucket setup for static hosting and a cloudfront distribution for serving the files. The problem is when I go to a subpage and refresh the page or go to a url directly I get a 403 error. This does not happen when I append /index.html at the end of the url.

In my cloudfront settings I have default root object set to index.html so according to the documentation everything should work. This is not a SPA, so setting 403 error page to index.html won't help - it will just redirect to home page.

The desired behaviour I'm after is when user goes to url http://example.com/path/to/directory, index.html file from /path/to/directory/index.html is served.

S3 Bucket permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::[bucket-name]/*"
        },
        {
            "Sid": "2",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::cloudfront:user/[user]"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::[bucket-name]/*"
        },
        {
            "Sid": "3",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::cloudfront:user/[user]"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::[bucket-name]/*"
        }
    ]
}