How to read parquet file from s3 using pandas

Viewed 18

I am trying to read the parquet file which is in s3 using pandas.

Below is the code

import boto3
import pandas as pd



key = 'key'
secret = 'secret'
s3_client = boto3.client(
    's3',
    aws_access_key_id = key,
    aws_secret_access_key = secret,
    region_name = 'region_name'
)
print(s3_client)

AWS_S3_BUCKET='bucket_name'
filePath='data/wine_dataset'

response = s3_client.get_object(Bucket=AWS_S3_BUCKET, Key=filePath)

status = response.get("ResponseMetadata", {}).get("HTTPStatusCode")

if status == 200:
    print(f"Successful S3 get_object response. Status - {status}")
    books_df = pd.read_parquet(response.get("Body"))
    print(books_df)
else:
    print(f"Unsuccessful S3 get_object response. Status - {status}")

I am getting the below error

NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.

But when I read the same s3 path using pyspark it worked

path= 's3a://bucket_name/data/wine_dataset'

df = spark.read.parquet(path)

I am not sure why it is not working using pandas. Can anyone help me on this?

0 Answers
Related