org.apache.spark.sql.AnalysisException: resolved attribute(s)

user_8048750 0
Viewed 3039

Normally I work with Spark on Scala API, however I have to write something in pyspark, and getting error that is difficult to troubleshoot.

I am doing a join on 3 DFs :

df_c = gr_o.select("col1", "col2", "col3", "col4").join(gr_r.select("col1", "col5"), "col1", how='left').join(gr_v.select("col1", "col6"), "col1", how='left')

And when I do df_c.show() getting following error:

py4j.protocol.Py4JJavaError: An error occurred while calling o724.join.
: org.apache.spark.sql.AnalysisException: resolved attribute(s)......
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 20, in fun
  File "/spark/python/pyspark/sql/dataframe.py", line 828, in join
    jdf = self._jdf.join(other._jdf, on, how)
  File "/spark/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
  File "/spark/spark/python/pyspark/sql/utils.py", line 69, in deco
    raise AnalysisException(s.split(': ', 1)[1], stackTrace)
pyspark.sql.utils.AnalysisException: 'resolved attribute(s)

Source for DF is a VIEW, when I use a table instead of VIEW then it works. I am wondering if someone has experienced similar issue.

pyspark apache-spark
2 Answers

Ive thought to put here solution what I used, maybe someone will find it useful. I still dont know why it doesnt work as supposed to, looks like a bug in pyspark. Basically I put alias for a column in one dataframe, alias has the same name (col1 as col1) what the actual column name.

df_c = (
    gr_o
    .select("col1", "col2", "col3", "col4")
    .join(gr_r.selectExpr("col1 as col1", "col5"), "col1", how='left')
    .join(gr_v.select("col1", "col6"), "col1", how='left')
)
user_80487500

You should always break down your data-frame likewise for a better readability for other developers in your production code. This is help simplify the debugging and understanding Now, coming to the problem, this looks like some column related mismatch..

gr_o = gr_o.select("col1", "col2", "col3", "col4")
gr_r = gr_r.select("col1", "col5")
gr_v = gr_v.select("col1", "col6")

df_c = gr_o.join(gr_r,"col1","left")
df_c = df_c.join(gr_v,"col1","left")
user_51082200
Related