Unable to create Hive tables in pyspark inside google Colab using hivesupportenabled?

Viewed 295

Here is my code: -

from pyspark import SparkContext, SparkConf
from pyspark.conf import SparkConf
from pyspark.sql import SparkSession, HiveContext
from pyspark.sql import SparkSession
sparkSession = (SparkSession
                .builder
                .master("local")
                .appName('Colab')
                .config('spark.ui.port', '4050')
                .enableHiveSupport()
                .getOrCreate()
                )

sc.createOrReplaceTempView("my_temp_table");
spark.sql("drop table if exists my_table");
spark.sql("CREATE TABLE Starcraft AS SELECT * FROM my_temp_table")

I tried changing

master("local")

with

also tried adding

    .config("spark.sql.catalogImplementation","hive")\

but to no avail.

The error:-

AnalysisException: Hive support is required to CREATE Hive TABLE (AS SELECT);;
'CreateTable `Starcraft`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, ErrorIfExists
+- Project [Name#3094, Dept#3095, Salary#3096]
   +- SubqueryAlias my_temp_table
      +- Relation[Name#3094,Dept#3095,Salary#3096] csv
    master("yarn")

Is it a problem of the environment or the code?

0 Answers
Related