Any fix for UserWarning: pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream?

user_7431349 0
Viewed 2202

On converting spark df to pandas df using pyarrow function i am getting following warning:

UserWarning: pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream

I am using python 3.7 version and Pyspark 2.4.3 pyspark df size is 170000 rows and 40 columns On converting it to pandas using Pyarrow function i am getting only 61585 rows and 40 columns as output despite getting complete data.

train_set.count()
170000

spark.conf.set("spark.sql.execution.arrow.enabled", "True")
result_pdf = train_set.select("*").toPandas()

> C:\anaconda\lib\site-packages\pyarrow\__init__.py:152: UserWarning:
> pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream 
> warnings.warn("pyarrow.open_stream is deprecated, please use "


 result_pdf.shape
    (61585, 40)

Expected:

result_pdf.shape
(170000,40)

Actual:

result_pdf.shape
(61585,40)
pyspark pyarrow pandas apache-spark
1 Answers

Finally I found a solution for the above query. It was a datatype issue. I n one of my column I was generating probability while processing in spark which was giving output as 4.333333 Incase probability is 4.3 and post rounding it off was also not working because while converting itself it was not storing all the rows for above column. Since spark process it using Java ,there is predefined length of float and string variable which java allows to process. The above column was crossing same. Hence it was storing the df with some of the rows in error form. On removing this column when I tried to convert in pandas it converted all rows swiftly.

user_74313490
Related