What are best practices to migrate parquet table to Delta?

user_15546071 0
Viewed 108

I'm trying to convert large parquet files to delta format for performance optimization and a faster job run.

I'm trying to research the best practices to migrate huge parquet files to delta format on Databricks.

delta-lake databricks
1 Answers

There are two general approaches to that, but it's really depends on your requirements:

  1. Do in-place upgrade using the CONVERT TO DELTA (SQL Command) or corresponding Python/Scala/Java APIs (doc). You need to take into account following consideration - if you have a huge table, then default CONVERT TO DELTA command may take too long as it will need to collect statistics for your data. You can avoid this by adding NO STATISTICS to the command, and then it will run faster. With it, you won't be able to get benefits of data skipping, and other optimizations, but these statistics could be collected later when executing OPTIMIZE command.

  2. Create a copy of your original table by reading original Parquet data & writing as a Delta table. After you check that everything is correct, you may remove original table. This approach have following benefits:

    • You can change partitioning schema if you have too many levels of partitioning in your original table
    • You can change the order of columns in the table to take advantage of data skipping for numeric & date/time data types - it should improve the query performance.
Alex_Ott_186270
Related