I am trying two ways of filling out missing value, one works the other doesn't. (i'm on google colab), with following version of pyspark
!pip install pyspark==3.0.1 py4j==0.10.9
Code works:
data.na.fill(value=0, subset=["data", "open"]).na.fill(value='unknown', subset=["high"]).show(5)
Code that not work
data.na.fill({'high':0}).show(5)
Returns me error:
/usr/local/lib/python3.7/dist-packages/pyspark/sql/utils.py in raise_from(e)
AnalysisException: Cannot resolve column name "market.cap" among (_c0, symbol, data, open, high, low, close, volume, adjusted, market.cap, sector, industry, exchange); did you mean to quote the `market.cap` column?;
A friendly guest left a comment just now (but then quickly deleted for some reason) that the error is due to the col name market.cap. Thank him for pointing it out. I should be more clearer about my major concern here. Why the market.cap col causing issue with the 2nd line of the command but not the first line of command. And how may I fix this col name?