I have a Spark application that loads CSV files, converts it to a Parquet file, store Parquet file in my Data Lake storage, and then load data into a BigQuery table.
The problem is when CSV has too much old timestamp value, the conversion occurs but the timestamp column can't be shown in a BigQuery table.
When I set the config spark.sql.parquet.outputTimestampType to TIMESTAMP_MICROS, I got this error on BigQuery:
Cannot return an invalid timestamp value of -62135607600000000 microseconds relative to the Unix epoch. The range of valid timestamp values is [0001-01-1 00:00:00, 9999-12-31 23:59:59.999999]; error in writing field reference_date
When I set the config spark.sql.parquet.outputTimestampType to TIMESTAMP_MILLIS, I got this error on Airflow:
Error while reading data, error message: Invalid timestamp value -62135607600000 for field 'reference_date' of type 'INT64' (logical type 'TIMESTAMP_MILLIS'): generic::out_of_range: Invalid timestamp value: -62135607600000
- CSV file:
id,reference_date
"6829baef-bcd9-412a-a2f3-abdfed02jsd","0001-01-02 21:00:00"
- Read the CSV file (and convert
reference_dateto a Timestamp column):
def castDFColumn(
df: DataFrame,
column: String,
dataType: DataType
): DataFrame = df.withColumn(column, df(column).cast(dataType))
...
var df = spark
.read
.format("csv")
.option("header", true)
.load("myfile.csv")
df = castDFColumn(df, "reference_date", TimestampType)
- Convert to a file Parquet file:
df
.write
.mode("overwrite")
.parquet("path/to/save")
- Spark application runtime configurations:
val conf = new SparkConf().setAppName("Load CSV")
conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
conf.set("spark.sql.parquet.outputTimestampType", "TIMESTAMP_MILLIS/TIMESTAMP_MICROS")
conf.set("spark.sql.session.timeZone", "UTC")
It looks like the timestamp is changed to 0000-12-31 21:00:00, or something like that, getting out of the acceptable range of INT64 timestamp.
Has anyone been through this?