parse multiple parquet files with s3 select in python?

user_5752202 0
Viewed 1612

I'm trying to read multiple parquet files from a single S3 bucket subfolder with boto3.

I've had no problem reading a single csv file with python, but I have'nt been able to get it to work with multiple file readings before.

I have seen previous answers that this is not supported by aws. However, as I understand, this functionality has been implemented in the end of 2018.

So, is there a way, using my working code (shown below) to run a s3 select statement for all the parquet files in the relevant folder, i.e. select all the rows from all the files?

Files in the working subfolder in a bucket:

_success
file1.snapy.parquet
file2.snapy.parquet

python code:

response = s3.select_object_content(
            Bucket='somebucket',
            Key= 'pathtosubfolder',
            ExpressionType='SQL', <br>
            InputSerialization = {'Parquet': {}},
            Expression="select * from s3object s ",
            OutputSerialization = {'CSV': {}},
            )

I expected the sql statement outcome, however I'm getting:

[ERROR] NoSuchKey: An error occurred (NoSuchKey) when calling the SelectObjectContent operation: The specified key does not exist.

parquet amazon-s3
1 Answers

You probably want to take a look at Amazon Athena if you want to query collections of parquet format objects in an S3.

Athena is fully managed Facebook Presto service.

user_19712150
Related