Databricks Delta storage - Caching tables for performance

user_3110297 0
Viewed 448

While investigating ways of trying to improve the performance of some queries I bumped into Delta storage Cache options, it has left me with several questions. (a little knowledge is dangerous)

spark.conf.set("spark.databricks.io.cache.enabled", "true")

  • Does the code above affect just the notebook I'm in, or does it apply to the cluster.
  • If it is the cluster does it resume after the cluster has been restarted?

cache select * from tablename

  • Does the code above cache the table contents and can be benefitted from if I then do a select on 1 column and join to another table? or does the cache only operate if that exact command is issued again (select * from tablename)?

I've basically got 3 tables that will be used a lot for analysis and I wanted to improve performance. I've created them as Delta storage, partitioned on columns I think are likely to be most commonly used for filtering clauses (but not too high cardinality), and applied zorder on a column that matches all 3 tables and will be used in all joins between them. I'm now exploring caching options to see if I can improve performance even more.

databricks apache-spark caching performance
1 Answers

See https://docs.databricks.com/delta/optimizations/delta-cache.html

In short:

  • It applies to your cluster and has nothing to do with your notebook.

  • It does not support CSV, JSON, and ORC.

  • Your choice of cluster config can affect the setup and operation. See URI.

  • You can use Delta caching and Apache Spark caching at the same time. E.g. the Delta cache contains local copies of remote data. It can improve the performance of a wide range of queries, but cannot be used to store results of arbitrary subqueries. That is what Spark caching is for.

user_69339930
Related