Pyspark column selection method difference

user_9261745 0
Viewed 30

I am kind of confused about col function. I know that I can use the following methods to select a column

  • df.select(df.key)
  • df.select(col("key"))
  • df.select(df["key"])
  • df.select("key")

So, what are their differences?

from pyspark.sql import functions as F

df = spark.createDataFrame([("A", 1), ("B", 2), ("C", 3)], ["key", "value"])

df.show()
+---+-----+                                                                     
|key|value|
+---+-----+
|  A|    1|
|  B|    2|
|  C|    3|
pyspark
1 Answers

df.select(df.key)
  df: modify when dataframe name is changed
  key: no space, no special character

df.select(col(key))
  col(): need import pyspark.sql.functions.col

df.select(df[key])
  df: modify when dataframe name is changed

df.select(key)
  no problem

user_174756670
Related