How to insert csv data into an existing SQL table

Viewed 301

I have created a database and a table (table1) using an SQL syntax and execute them using spark.sql :

spark.sql("CREATE TABLE table1...");

I also loaded a csv file data into a dataframe using :

Dataset<Row> firstDF = spark.read().format("csv").load("C:/file.csv");

Now i use the following code to populate the existing table with the csv data :

firstDF.toDF().writeTo("table1").append();

But when i select all from the table1 :

Dataset<Row> firstDFRes = spark.sql("SELECT * FROM table1");
firstDFRes.show();

i get it empty (with only the schema of the table and no data) My question is how to populate an existing SQL table with a dataframe ?

PS : using DataFrameReader's InsertInto or else SaveAsTable will create the table using the csv data and will ignore the schema of the SQL created table.

Thank you.

0 Answers
Related