How can I allow precision loss over pyspark jdbc from oracle?

Viewed 353

I am attempting to move a very large Oracle table which contains 300+ columns and 10,000,000+ records using pyspark 3.0.1 over DataFrameReader.jdbc. I get the error message:

java.lang.ArithmeticException: Decimal precision 136 exceeds max precision 38

  • I can't say which record, or even which column for that matter, is causing this error.
  • Furthermore, I plan to repeat this process with many tables and need to automate handling this error.
  • I do not have the permissions to change any data in the source database (read-only).
  • Decimal precision of 136 is not necessary for my use cases.
  • If there is a way to actually allow precision of 136, I would also be ok with that solution.

Everything I find online about this issue is regarding others wanting to preserve their decimal precision (not reduce it). I have tried adding the following to my SparkSession config:

spark.sql.decimalOperations.allowPrecisionLoss=true

But I still get the same error.

Update:

I was able to get the data to load, then I cast the decimal columns to float.

raw_data = self._get_spark().read.jdbc(self._url, self._table, **self._load_args)
decimal_cols = []
for column in raw_data.columns:
    if isinstance(raw_data.schema[column].dataType, DecimalType):
        decimal_cols.append(column)

    for col_name in decimal_cols:
        raw_data = raw_data.withColumn(col_name, col(col_name).cast('float'))

Next, I add a new column and make it a float as well. Printing data.schema shows that none of my datatypes in the dataframe is DecimalType. Finally, whenever I go to write back over jdbc I get the same error message.

data.write.jdbc(self._url, self._table, **self._save_args)

java.lang.ArithmeticException: Decimal precision 136 exceeds max precision 38

0 Answers
Related