I got a warning by setting spark.sql.optimizer.metadataOnly to be true. I use this property mainly for querying partition columns without reading the parquet files. It is very useful to figure which partitions I want to read, for example getting distinct values from a partition column very quickly.
How would SparkSessionExtensions work to keep this property in the future or is there an alternative way to query partition columns without reading the data files? (I code in pyspark)
WARN SQLConf: The SQL config 'spark.sql.optimizer.metadataOnly' has been deprecated in Spark v3.0 and may be removed in the future. Avoid to depend on this optimization to prevent a potential correctness issue. If you must use, use 'SparkSessionExtensions' instead to inject it as a custom rule.