What's the difference between spark.sql() and spark.read.format("jdbc").option("query", "")?

user_11734835 0
Viewed 1444

I assumed that spark.sql(query) is used when we are using spark sql and that spark.read.format("jdbc").option("query", "") is used when we are using oracle sql syntax. Would I be right in assuming so?

pyspark apache-spark-sql apache-spark python
1 Answers

Yes and Spark does more than that too!

Spark-Jdbc:

From Spark docs Jdbc(Java Database connectivity) is used to read/write data from other databases (oracle, mysql, sqlserver, postgres, db2..etc).

spark.read.format("jdbc").option("query", "(select * from <db>.<tb>)e")

Spark-Sql:

From docs Spark's module for working with structured data and lets you to query data using DataFrame API or SQL API.

  1. We can use Spark-Sql to read data from hive/presto/athena/delta/csv/parquet...etc files.
  2. Create a temporary view/table on dataframe then run SQL queries.
  3. Easily write RDD's/Dataframes to Hive or Parquet files.

enter image description here

user_76326950
Related