Unable to read parquet file using StreamingBody from S3 without holding in memory

I'm trying to read a parquet file from S3, and dump the contents of it on to a Kafka topic.

This isn't too difficult when you hold the entire file in memory, but for large files this isn't feasible.

# using .read() holds the entire file open in memory - not ideal
df = pd.read_parquet(s3_response['Body'].read(), columns=columns)

Instead, I'm trying to take advantage of file-like-objects in order to stream the parquet file.

My issue is that it seems that it's impossible to do this with Parquet, because Parquet encodes data in the footer of the file as well as the header.

Here's an example of my code:

session = boto3.session.Session()

s3_client = session.client(
    service_name='s3',
    endpoint_url=s3_url,
)

obj = s3_client.get_object(Bucket=s3_bucket, Key=key)

for line in obj['Body'].iter_lines():
    pq_file = io.BytesIO(line)
    df = pd.read_parquet(pq_file, columns=columns)
    # At this point I'd want to iterate over the DF rows
    # and send them to kafka
    print(df)

This results in the following error: OSError: Could not open parquet input source '<Buffer>': Invalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.

Is it at all possible to do what I'm trying to do? Or due to the nature of parquet files is this impossible?