Here is my code: -
from pyspark import SparkContext, SparkConf
from pyspark.conf import SparkConf
from pyspark.sql import SparkSession, HiveContext
from pyspark.sql import SparkSession
sparkSession = (SparkSession
.builder
.master("local")
.appName('Colab')
.config('spark.ui.port', '4050')
.enableHiveSupport()
.getOrCreate()
)
sc.createOrReplaceTempView("my_temp_table");
spark.sql("drop table if exists my_table");
spark.sql("CREATE TABLE Starcraft AS SELECT * FROM my_temp_table")
I tried changing
master("local")
with
also tried adding
.config("spark.sql.catalogImplementation","hive")\
but to no avail.
The error:-
AnalysisException: Hive support is required to CREATE Hive TABLE (AS SELECT);;
'CreateTable `Starcraft`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, ErrorIfExists
+- Project [Name#3094, Dept#3095, Salary#3096]
+- SubqueryAlias my_temp_table
+- Relation[Name#3094,Dept#3095,Salary#3096] csv
master("yarn")
Is it a problem of the environment or the code?