pyspark.sql.functions.col selecting not existing column

Viewed 314

I just encurred a strange behavior on pyspark.sql.functions.col.

Filtering and selecting part of dataframe, I had a correct output less column than expected. I followed the code and I found that I was filtering on a column that was not selected previously, but I had no error selecting it with sql.functions.col method.

I can't understand with PySpark is not raising an exception with pyspark.sql.functions.col on a not existing column. Moreover, it has the correct behaviour as the column was present.

Let me explain it better with an example in python:

df = library.spark.createDataFrame(
    [
        (1, "foo"),
        (2, "bar"),
    ],
    ["Id", "Label"]
)
df.show(truncate=False)
Id Label
1 foo
2 bar
df = df.select(F.col("Id"))
df.show(truncate=False)
Id
1
2
df = df.filter(F.col("Label") == "foo")
df.show(truncate=False)
Id
1

I was wondering on how is it possible that pyspark filters the the df dataframe on a column that is not selected with the previous select.

I just have done some tentative and I found that the issue is caused by pyspark.sql.functions.col since if we do the filter in the following ways, an exception is correctly fired, as I was expected.

df = df.filter(df.Label == "foo")

with AttributeError

df = df.filter(df["Label"] == "foo")

with AnalysisException

Can someone explain me how is sql.col working in this piece of code?

0 Answers
Related