How to update or refresh ddl in Spark automatically from MySQL

Viewed 55

I'm using spark as datalake, the data source are tables from MySQL.

Let's say there is a table in MySql. name and lastname are int data types

enter image description here

Then in Spark-sql

CREATE TABLE IF NOT EXISTS super_hero
USING org.apache.spark.sql.jdbc
OPTIONS (
  url "jdbc:mysql://XXX.XXX.YYYY.YYYY:3306/test?useSSL=FALSE&nullCatalogMeansCurrent=true&zeroDateTimeBehavior=convertToNull",
  driver "com.mysql.cj.jdbc.Driver",
  dbtable "test.super_hero",
  user "USER",
  password "PASSWORD"
);

It works fine in spark-sql I ran:

    spark-sql> select * from super_hero;
1       0       0        superX        111-111

Then if I change the data types in MySql and then I re-ran the same command in Spark-sql

spark-sql> select * from super_hero;
22/07/25 22:57:05 ERROR Executor: Exception in task 0.0 in stage 4.0 (TID 4)
java.sql.SQLDataException: Cannot determine value type from string 'Luis'
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:114)
...
Caused by: com.mysql.cj.exceptions.DataConversionException: Cannot determine value type from string 'Luis'
        at com.mysql.cj.result.AbstractNumericValueFactory.createFromBytes(AbstractNumericValueFactory.java:65)

Is there any configuration to create a table in spark-sql to automictically get/refresh the latest DDL and data from MySQL?

So far I have drop the table in spark-sql and recreate the table to get the latest ddl and data (in case of new columns).

0 Answers
Related