Encoding issue using pd.read_csv() to read csv file in S3 location

user_16042606 0
Viewed 772

Problem: I am getting an encoding error while trying to use pd.read_csv() to read a CSV file in an S3 location.

Below is my code:

 # parameters
 s3_bucket = 'my_bucket'
 s3_key = 'my_key'

 # create s3 client
 s3_client = boto3.client('s3')

 # create s3 object
 obj = s3_client.get_object(Bucket=s3_bucket, Key=s3_key)
   
 # read csv file from s3
 df = pd.read_csv(obj['Body'], encoding='cp1252')

But this is the error I get: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

I don't understand why I am getting the encoding error when I am specifying the encoding to be 'cp1252'. By the way, 'cp1252' is the encoding I found for my csv file.

boto3 pandas amazon-s3 python
1 Answers

I looked into the boto3 documentation and the get_object() method returns a StreamBody. The pandas method read_csv() takes a path, file, buffer and so on as input (documentation).

Therefore, I think you have to convert the object body first. This can be done with Python's io module (documentation). The following code should fix your problem:

obj = s3_client.get_object(Bucket=s3_bucket, Key=s3_key)
df = pd.read_csv(io.BytesIO(obj['Body'].read()))

Explanation: Pandas states in the doc:

By file-like object, we refer to objects with a read() method, such as a file handle (e.g. via builtin open function) or StringIO.

This is fulfilled by giving the StreamBody to io.BytesIO from which you can read the bytes of your file.

user_79479940
Related