Manifest for multiple JSON files in S3

Question

I am trying to create a quicksight data source using a manifest for S3. The data at S3 is multiple JSON documents. Millions of them. I can create a dataset using a single document like this:

{
    "fileLocations": [
        {
            "URIs": [
                "https://s3-us-west-2.amazonaws.com/my-bucket/doc1.json"
            ]
        }
    ],
    "globalUploadSettings": {
        "format": "JSON"
    }
}

However, I would like ALL of the JSON documents. I am trying to do this without success:

{
    "fileLocations": [
        {
            "URIs": [
                "https://s3-us-west-2.amazonaws.com/my-bucket/*.json"
            ]
        }
    ],
    "globalUploadSettings": {
        "format": "JSON"
    }
}

How can you create a wildcard manifest for JSON documents on S3?

user_5151350 · Answer

The following should get all files in a bucket

{
    "fileLocations": [
        {
            "URIPrefixes": [ 
                "s3://my-bucket/"          
            ]
        }
    ],
    "globalUploadSettings": {
        "format": "JSON"
    }
}

Manifest for multiple JSON files in S3

1 Answers