Spark 2.3+ use of parquet.enable.dictionary?

user_2507406 0
Viewed 4815

I am looking for documentation on how parquet.enable.dictionary is to be used in Spark (latest 2.3.1). It can be set to "true" or "false" when creating a SparkSession.

I googled for any documentation on this feature and found nothing, or at least nothing recent.

Specifically these are my questions:

Is parquet.filter.dictionary.enabled = true or = false by default in Spark 2.3.1?

Is this a feature to enable (set to true) before I write to Parquet files so that Parquet library of Spark computes and writes the dictionary information to disk?

Is this setting ignored when Spark reads the Parquet files or do I still need to set it to true for reading parquet (as well as writing)?

When should I use this feature (set to true) ? Pros/Cons?

I also see references to this spark.hadoop.parquet.enable.dictionary when I googled for the parquet.enable.dictionary. Is this related? Which should I use?

Are there any other Spark + Parquet settings I need to be aware of?

Many thanks!

parquet apache-spark
2 Answers

These are the spark parquet config set to false by default -

spark.sql.parquet.mergeSchema
spark.sql.parquet.respectSummaryFiles
spark.sql.parquet.binaryAsString
spark.sql.parquet.int96TimestampConversion
spark.sql.parquet.int64AsTimestampMillis
spark.sql.parquet.writeLegacyFormat
spark.sql.parquet.recordLevelFilter.enabled

Below are set to true by default -

spark.sql.parquet.int96AsTimestamp
spark.sql.parquet.filterPushdown
spark.sql.parquet.filterPushdown.date
spark.sql.parquet.filterPushdown.timestamp
spark.sql.parquet.filterPushdown.decimal
spark.sql.parquet.filterPushdown.string.startsWith
spark.sql.parquet.enableVectorizedReader

These properties needs value and listing it with defaults-

spark.sql.parquet.outputTimestampType = INT96
spark.sql.parquet.compression.codec = snappy
spark.sql.parquet.pushdown.inFilterThreshold = 10
spark.sql.parquet.output.committer.class = org.apache.parquet.hadoop.ParquetOutputCommitter
spark.sql.parquet.columnarReaderBatchSize = 4096

Regarding parquet.enable.dictionary, it is not supported by Spark yet. But it can be set in sqlContext as -

sqlContext.setConf("parquet.enable.dictionary", "false")

Default value is of this property is true in parquet. Therefore, it should be true when parquet code is called from Spark.

user_86174630

This can help too:

Parquet file properties are set at write time. The properties listed in Table 13-3 (parquet.block.size, parquet.page.size, parquet.dictionary.page.size, parquet.enable.dictionary) are appro-priate if you are creating Parquet files from MapReduce (using the formats discussed inParquet MapReduce), Crunch, Pig, or Hive.

http://what-when-how.com/Tutorial/topic-2059e313/Hadoop-The-Definitive-Guide-457.html

user_20215160
Related