How to extend Spark Catalyst optimizer with custom rules?

user_2764858 0
Viewed 2050

I want to use Catalyst rules to transform star-schema (https://en.wikipedia.org/wiki/Star_schema) SQL query to SQL query to denormalized star-schema where some fields from dimensions tables are represented in facts table. I tried to find some extension points to add own rules to make a transformation described above. But I didn't find any extension points. So there are the following questions:

  1. How can I add own rules to catalyst optimizer?
  2. Is there another solution to implement a functionality described above?
apache-spark-sql apache-spark
2 Answers

Following @Ambling advice you can use the sparkSession.experimental.extraStrategies to add your functionality to the SparkPlanner.

An example strategy that simply prints "Hello world" on the console

object MyStrategy extends Strategy {
  def apply(plan: LogicalPlan): Seq[SparkPlan] = {
    println("Hello world!")
    Nil
  }
}

with example run:

val spark = SparkSession.builder().master("local").getOrCreate()

spark.experimental.extraStrategies = Seq(MyStrategy)
val q = spark.catalog.listTables.filter(t => t.name == "five")
q.explain(true)
spark.stop()

You can find a working example project on friend's GitHub: https://github.com/bartekkalinka/spark-custom-rule-executor

user_15491350
Related