How to bucket tables using AWS Glue and Spark SQL?

Viewed 621

I am trying to run this query on AWS Glue

CREATE TABLE bucketing_example
  USING parquet
  CLUSTERED BY (id) INTO 2 BUCKETS
  LOCATION 's3://my-bucket/bucketing_example'
  AS SELECT * FROM (
   VALUES(1, 'red'),
         (2, 'orange'),
         (5, 'yellow'),
         (10, 'green'),
         (11, 'blue'),
         (12, 'indigo'),
         (20, 'violet'))
   AS Colors(id, value)

and I am getting the following exception:

java.lang.IllegalArgumentException: Can not create a Path from an empty string
  at org.apache.hadoop.fs.Path.checkPathArg(Path.java:163)
  at org.apache.hadoop.fs.Path.<init>(Path.java:175)
  at org.apache.spark.sql.catalyst.catalog.CatalogUtils$.stringToURI(ExternalCatalogUtils.scala:236)
  at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getDatabase$1$$anonfun$apply$2.apply(HiveClientImpl.scala:343)
  at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getDatabase$1$$anonfun$apply$2.apply(HiveClientImpl.scala:339)
  at scala.Option.map(Option.scala:146)

Also, I tried to run a Spark SQL query similar to these on a bucketed table that was created using Athena (still using Glue).

Although the a DESCRIBE EXTENDED on the table reveals the bucket column, the Exchanges on the arms of the JOIN remain in the plan.

Does bucketing work with Glue and Spark SQL?

1 Answers
Related