Reading a feather file from S3 using api gateway proxy for S3

Viewed 105

I am trying to read a feather file from an S3 bucket using api gateway proxy for s3 with no luck. Tried everything but every time I get below error.

ine 239, in read_table
    reader = _feather.FeatherReader(source, use_memory_map=memory_map)
  File "pyarrow\_feather.pyx", line 75, in pyarrow._feather.FeatherReader.__cinit__
  File "pyarrow\error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow\error.pxi", line 114, in pyarrow.lib.check_status
OSError: Verification of flatbuffer-encoded Footer failed.

I am able to read the same feather file from my local. So looks like nothing is wrong with the feather file itself. It has something to do with how api gateway is returning reponse.

Tried to import this openapi specification for api gateway to get rid of any issues but still same error.

Can anyone please guide me here.

Python code #1

import boto3
from io import BytesIO 
import pandas as pd

s3_data=pd.read_feather("https://<api_gateway>/final/s3?key=naxi143/data.feather")

Python code #2

import pandas as pd
import requests
import io

header={'Accept': 'application/octet-stream'}
resp = requests.get(
    'https://<api_gateway>/final/s3?key=naxi143/data.feather',
    stream=True,
    headers=header
)

#print(resp.json())
resp.raw.decode_content = True
mem_fh = io.BytesIO(resp.raw.read())
print(mem_fh )
pd.read_feather(mem_fh)
0 Answers
Related