I am attempting to move a very large Oracle table which contains 300+ columns and 10,000,000+ records using pyspark 3.0.1 over DataFrameReader.jdbc. I get the error message:
java.lang.ArithmeticException: Decimal precision 136 exceeds max precision 38
- I can't say which record, or even which column for that matter, is causing this error.
- Furthermore, I plan to repeat this process with many tables and need to automate handling this error.
- I do not have the permissions to change any data in the source database (read-only).
- Decimal precision of 136 is not necessary for my use cases.
- If there is a way to actually allow precision of 136, I would also be ok with that solution.
Everything I find online about this issue is regarding others wanting to preserve their decimal precision (not reduce it). I have tried adding the following to my SparkSession config:
spark.sql.decimalOperations.allowPrecisionLoss=true
But I still get the same error.
Update:
I was able to get the data to load, then I cast the decimal columns to float.
raw_data = self._get_spark().read.jdbc(self._url, self._table, **self._load_args)
decimal_cols = []
for column in raw_data.columns:
if isinstance(raw_data.schema[column].dataType, DecimalType):
decimal_cols.append(column)
for col_name in decimal_cols:
raw_data = raw_data.withColumn(col_name, col(col_name).cast('float'))
Next, I add a new column and make it a float as well. Printing data.schema shows that none of my datatypes in the dataframe is DecimalType. Finally, whenever I go to write back over jdbc I get the same error message.
data.write.jdbc(self._url, self._table, **self._save_args)
java.lang.ArithmeticException: Decimal precision 136 exceeds max precision 38