I'm using spark as datalake, the data source are tables from MySQL.
Let's say there is a table in MySql. name and lastname are int data types
Then in Spark-sql
CREATE TABLE IF NOT EXISTS super_hero
USING org.apache.spark.sql.jdbc
OPTIONS (
url "jdbc:mysql://XXX.XXX.YYYY.YYYY:3306/test?useSSL=FALSE&nullCatalogMeansCurrent=true&zeroDateTimeBehavior=convertToNull",
driver "com.mysql.cj.jdbc.Driver",
dbtable "test.super_hero",
user "USER",
password "PASSWORD"
);
It works fine in spark-sql I ran:
spark-sql> select * from super_hero;
1 0 0 superX 111-111
Then if I change the data types in MySql and then I re-ran the same command in Spark-sql
spark-sql> select * from super_hero;
22/07/25 22:57:05 ERROR Executor: Exception in task 0.0 in stage 4.0 (TID 4)
java.sql.SQLDataException: Cannot determine value type from string 'Luis'
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:114)
...
Caused by: com.mysql.cj.exceptions.DataConversionException: Cannot determine value type from string 'Luis'
at com.mysql.cj.result.AbstractNumericValueFactory.createFromBytes(AbstractNumericValueFactory.java:65)
Is there any configuration to create a table in spark-sql to automictically get/refresh the latest DDL and data from MySQL?
So far I have drop the table in spark-sql and recreate the table to get the latest ddl and data (in case of new columns).
