PyArrow Table to PySpark Dataframe conversion

user_9629771 0
Viewed 2176

I am trying to convert my Pandas dataframe to a PySpark dataframe. The createDataFrame function doesn't work, so I've found PyArrow.

I can convert Pandas --> a PyArrow table, but I can't find any way to convert PyArrow table --> Spark

Does anyone have any idea how I may achieve this?

Thanks

pyarrow pandas python-3.x
1 Answers

Late reply to this question, however, it keeps coming up in my search so I figure I will provide a method that I have used and maybe it will help other users.

I am using pyarrow to read a parquet file from an s3 object and then converting it to a pandas dataframe. I can then convert this pandas dataframe using a spark session to a spark dataframe.

import boto3
import pandas as pd
import io
import pyarrow.parquet as pq
from pyspark.context import SparkContext
from pyspark.sql.session import SparkSession

sc = SparkContext('local') #Pyspark normally has a spark context (sc) configured so this may not be necessary, use this if calling a python script using spark-submit
spark = SparkSession(sc) #This is used to convert the pandas dataframe to a spark dataframe

s3 = boto3.resource(/
service_name='s3',/
use_ssl=False,/
aws_access_key_id='S3_ACCESS_KEY_ID',/
aws_secret_access_key='S3_ACCESS_KEY',/
endpoint_url='S3_URL')

bucket_name = 'BUCKET_NAME'
object_name = 'OBJECT_NAME'

buffer = io.BytesIO()
s3_object = s3.Object(bucket_name,object_name)
s3_object.download_fileobj(buffer)
table = pq.read_table(buffer)
df = table.to_pandas()
df_spark = spark.createDataFrame(df) #**Conversion from pandas df to spark df**

All you need is a spark session to convert the pandas dataframe to a spark dataframe. I include the additional information for pyarrow since this post comes up when searching for pyarrow.

user_46523120
Related