I am trying to load pandas DataFrame to pyarrow Table with float column and have an issue during pyarrow Table construction: ArrowTypeError: ('int or Decimal object expected, got float', 'Conversion failed for column float_column with type object')
The code to reproduce:
import pandas as pd
import pyarrow as pa
from decimal import Decimal, Context
float_value = 1.2
print(type(float_value))
df = pd.DataFrame(data=[[float_value]], columns=["float_column"], dtype="object")
schema_dict = {"float_column":pa.decimal128(32, 6)}
schema = pa.schema(schema_dict)
table = pa.Table.from_pandas(df, schema)
The DataFrame itself is coming from pandas.read_sql_query method and the underlying type in the db is decimal(32,6).
The question is: How to specify SQL decimal(32,6) type in pandas DataFrame and convert such DataFrame to pyarrow Table?