I am trying to filter a Spark DataFrame (v. 1.5.0) and am seeing a peculiar result. First the results
df.groupby('behavior').count().show()
+---------+-------+
| behavior| count|
+---------+-------+
| Bought|1582345|
| Looked|2154789|
+---------+-------+
This is consistent with the number of rows in my data frame. Also, we see that there are only two "behaviors" in the column (Bought and Looked). Here's where things get weird.
df[df['behavior']=='Bought'].count()
1025879
df[df['behavior']=='Looked'].count()
698742
What happened? Where did the other rows of the data frame go? Finally, to make things even stranger, using the isin() method provides the correct results.
df[df['behavior'].isin(['Bought'])].count()
1582345
df[df['behavior'].isin(['Looked'])].count()
2154789
I have no clue what's going on here. I would have expected these two filters to at least return consistent results (both wrong or both right). Any help would be greatly appreciated!
Edit
Ran the following filter operations as suggested below, all results were consistent with the incorrect answer.
df.filter(df['behavior']=='Bought').count()
1025879
df.filter(df.behavior=='Bought').count()
1025879
df.filter(F.col('behavior')=='Bought').count()
1025879
So it seems like the equality check is what's wrong. What's interesting though is that the isin() functionality still seems to be working. I would have thought isin() used an equality check behind the scenes but if it is I don't know why it returns different results.