I just encurred a strange behavior on pyspark.sql.functions.col.
Filtering and selecting part of dataframe, I had a correct output less column than expected. I followed the code and I found that I was filtering on a column that was not selected previously, but I had no error selecting it with sql.functions.col method.
I can't understand with PySpark is not raising an exception with pyspark.sql.functions.col on a not existing column. Moreover, it has the correct behaviour as the column was present.
Let me explain it better with an example in python:
df = library.spark.createDataFrame(
[
(1, "foo"),
(2, "bar"),
],
["Id", "Label"]
)
df.show(truncate=False)
| Id | Label |
|---|---|
| 1 | foo |
| 2 | bar |
df = df.select(F.col("Id"))
df.show(truncate=False)
| Id |
|---|
| 1 |
| 2 |
df = df.filter(F.col("Label") == "foo")
df.show(truncate=False)
| Id |
|---|
| 1 |
I was wondering on how is it possible that pyspark filters the the df dataframe on a column that is not selected with the previous select.
I just have done some tentative and I found that the issue is caused by pyspark.sql.functions.col since if we do the filter in the following ways, an exception is correctly fired, as I was expected.
df = df.filter(df.Label == "foo")
with AttributeError
df = df.filter(df["Label"] == "foo")
with AnalysisException
Can someone explain me how is sql.col working in this piece of code?