Error when trying to write Pandas Dataframe to Parquet file

Question

I am trying to write a pandas Dataframe to a Parquet file.

Whenever i do this i get the following error:

pyarrow.lib.ArrowInvalid: ("Could not convert ' 10188018' with type str: tried to convert to int64", 'Conversion failed for column 1064 TEC serial with type object')

I have tried looking online and found some that had close to the same problem. Other solutions that did not help: https://discuss.streamlit.io/t/after-upgrade-to-the-latest-version-now-this-error-id-showing-up-arrowinvalid/15794 Tried to downgrade to streamlit 0.84.2 tried to change the config file tried to run the program like:

streamlit run app.py --global.dataFrameSerialization="legacy"

I also found the solution: ArrowTypeError: Did not pass numpy.dtype object', 'Conversion failed for column X with type int32

Did not work to downgrade or upgrade numpy

My code:

def saveAsParquet():
    print("Save as parquet")
    directory = askdirectory()
    directory = directory + "/"
    print(directory)
    for x in range (0,len(dataFrames)):
        filenameP = directory + fileNames[x] + ".parquet"
        filenameE = directory + fileNames[x] + ".xlsx"
        print(filenameP)
        print(filenameE)
        #dataFrames[x].dtypes.astype(str)
        dataFrames[x].to_parquet(path=filenameP,enige="fastparquet", compression="None")
        dataFrames[x].to_excel("C:\LogFIleCutter\LogfileTest.xlsx")
        print("Files exported")

I also tried the solution of turning the column in two string using the dtypes.astype(str)

Versions of stuff:

(base) C:\Windows\system32>pip show numpy
WARNING: Ignoring invalid distribution -ndas (c:\programdata\anaconda3\lib\site-packages)
WARNING: Ignoring invalid distribution -umpy (c:\programdata\anaconda3\lib\site-packages)
WARNING: Ignoring invalid distribution - (c:\programdata\anaconda3\lib\site-packages)
WARNING: Ignoring invalid distribution -andas (c:\programdata\anaconda3\lib\site-packages)
Name: numpy
Version: 1.19.0
Summary: NumPy is the fundamental package for array computing with Python.
Home-page: https://www.numpy.org
Author: Travis E. Oliphant et al.
Author-email:
License: BSD
Location: c:\programdata\anaconda3\lib\site-packages
Requires:
Required-by: tifffile, tables, streamlit, statsmodels, seaborn, scipy, scikit-learn, scikit-image, PyWavelets, pyerfa, pydeck, pyarrow, patsy, pandas, numexpr, numba, mkl-random, mkl-fft, matplotlib, imageio, imagecodecs, h5py, fastparquet, daal4py, Bottleneck, bokeh, bkcharts, astropy, altair

(base) C:\Windows\system32>streamlit version
Streamlit, version 0.84.2

Error when trying to write Pandas Dataframe to Parquet file

0 Answers