PySpark difference between pyspark.sql.functions.col and pyspark.sql.functions.lit

user_1259448 0
Viewed 34374

I find it hard to understand the difference between these two methods from pyspark.sql.functions as the documentation on PySpark official website is not very informative. For example the following code:

import pyspark.sql.functions as F
print(F.col('col_name'))
print(F.lit('col_name'))

The results are:

Column<b'col_name'>
Column<b'col_name'>

so what are the difference between the two and when should I use one and not the other?

pyspark apache-spark-sql
2 Answers

The doc says:

col:

Returns a Column based on the given column name.

lit:

Creates a Column of literal value

Say if we have a data frame as below:

>>> import pyspark.sql.functions as F
>>> from pyspark.sql.types import *

>>> schema = StructType([StructField('A', StringType(), True)])
>>> df = spark.createDataFrame([("a",), ("b",), ("c",)], schema)
>>> df.show()
+---+
|  A|
+---+
|  a|
|  b|
|  c|
+---+

If using col to create a new column from A:

>>> df.withColumn("new", F.col("A")).show()
+---+---+
|  A|new|
+---+---+
|  a|  a|
|  b|  b|
|  c|  c|
+---+---+

So col grabs an existing column with the given name, F.col("A") is equivalent to df.A or df["A"] here.

If using F.lit("A") to create the column:

>>> df.withColumn("new", F.lit("A")).show()
+---+---+
|  A|new|
+---+---+
|  a|  A|
|  b|  A|
|  c|  A|
+---+---+

While lit will create a constant column with the given string as the values.

Both of them return a Column object but the content and meaning are different.

user_49834500

To explain in a very succinct manner, col is typically used to refer to an existing column in a DataFrame, as opposed to lit which is typically used to set the value of a column to a literal

To illustrate with an example: Assume i have a DataFrame df containing two columns of IntegerType, col_a and col_b

  1. If i wanted a column total which were the sum of the two columns:
df.withColumn('total', col('col_a') + col('col_b'))
  1. Instead of i wanted a column fixed_val having the value "Hello" for all rows of the DataFrame df:
df.withColumn('fixed_val', lit('Hello'))
user_97980070
Related