I am trying to read a feather file from an S3 bucket using api gateway proxy for s3 with no luck. Tried everything but every time I get below error.
ine 239, in read_table
reader = _feather.FeatherReader(source, use_memory_map=memory_map)
File "pyarrow\_feather.pyx", line 75, in pyarrow._feather.FeatherReader.__cinit__
File "pyarrow\error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow\error.pxi", line 114, in pyarrow.lib.check_status
OSError: Verification of flatbuffer-encoded Footer failed.
I am able to read the same feather file from my local. So looks like nothing is wrong with the feather file itself. It has something to do with how api gateway is returning reponse.
Tried to import this openapi specification for api gateway to get rid of any issues but still same error.
Can anyone please guide me here.
Python code #1
import boto3
from io import BytesIO
import pandas as pd
s3_data=pd.read_feather("https://<api_gateway>/final/s3?key=naxi143/data.feather")
Python code #2
import pandas as pd
import requests
import io
header={'Accept': 'application/octet-stream'}
resp = requests.get(
'https://<api_gateway>/final/s3?key=naxi143/data.feather',
stream=True,
headers=header
)
#print(resp.json())
resp.raw.decode_content = True
mem_fh = io.BytesIO(resp.raw.read())
print(mem_fh )
pd.read_feather(mem_fh)