Operation not allowed: STORED BY hive on spark sql

user_11561121 0
Viewed 1361

I am trying to achieve a connection between Hive and DynamoDB via Spark. For that I need to execute this code:

sparkSession.sql("create external table test (id string) STORED BY 'org.apache.hadoop.hive.dynamodb.DynamoDBStorageHandler' TBLPROPERTIES('dynamodb.table.name'='testdb','dynamodb.column.mapping'='id:id')")

But I am having an error: 

ParseException: u"\nOperation not allowed: STORED BY(line 1, pos 111)

I read somewhere that STORED BY was not supported on Spark. Is there any alternative method? Is it always not supported? thank you

amazon-dynamodb apache-spark-sql apache-spark hive
1 Answers

Here is an excerpt from Spark Docs:

Note that, Hive storage handler is not supported yet when creating table, you can create a table using storage handler at Hive side, and use Spark SQL to read it.

As you can see, Spark cannot create Hive's StorageHandler tables, but can read already existing one.

The solution could be to create a table definition directly in Hive (using beeline, hive or any other JDBC client of your preference) and you will still be able to read the table in any Hive-enabled Spark (i.e., spark.sql("select * from test")).

user_52462110
Related