How can I detect if a colum contain constatnt values

user_7615872 0
Viewed 43

I want to write a script that returns the constant columns from a predefined dataframe. I got the idea of comparing each column min and max value, if they are equal I will append the column to a constant column list. This is how I'am doing it:

let's suppose I Have this dataframe:

>> df = spark.createDataFrame([(1., 4.,7.), (2., 5.,7.), (3., 6.,7.)], ["A", "B", "C"])
>> df.show()
+---+---+---+
|  A|  B|  C|
+---+---+---+
|1.0|4.0|7.0|
|2.0|5.0|7.0|
|3.0|6.0|7.0|
+---+---+---+

So here I have the "C" colum is constant as it contains the same value "7.0".

>> minA = df.agg({"A": "min"})
>> maxA = df.agg({"A": "max"})
>> minC = df.agg({"C": "min"})
>> maxC = df.agg({"C": "max"})

The next step is to compare in each case the min and max value: 

>>> maxA["max(A)"] == minA["min(A)"]
Column<(max(A) = min(A))>
>>> maxC["max(C)"] == minC["min(C)"]
Column<(max(C) = min(C))>

those results are meaningless, how do I solve this issue knowing that I dont want to use .collect() to retreive min and max value because this script has to work also in enormous amount of data and collect is not a good plan to go for it!

pyspark apache-spark
2 Answers

Do something like this:

from pyspark.sql.functions import min, max

[c for c, const in df
    .select([(min(c) == max(c)).alias(c) for c in df.columns])
    .first().asDict().items() if const]
user_32698090

I would go for something like this:

df = spark.createDataFrame([(1., 4.,7.), (2., 5.,7.), (3., 6.,7.)], ["A", "B", "C"])

# Trying to identify constant values in column C, so the index for the key is 2
rdd = df.map(lambda k: (k[2], (k[0], k[1]))).reduceByKey(lambda x, y: (0,0))
amount_of_different_elements = rdd.count()
user_35470320
Related