Estoy haciendo un ejercicio donde obtengo los datos de una columna en formato tabla, lo puedo hacer con la columna employees pero no con deparment
from pyspark.sql.functions import explode
explodeDF = comoquieradf.select(explode("employees").alias("e"))
flattenDF = explodeDF.selectExpr("e.firstName", "e.lastName", "e.email", "e.salary")
flattenDF.show()
output
+---------+--------+--------------------+------+
|firstName|lastName| email|salary|
+---------+--------+--------------------+------+
| michael|armbrust|no-reply@berkeley...|100000|
| xiangrui| meng|no-reply@stanford...|120000|
| matei| null|no-reply@waterloo...|140000|
| null| wendell|no-reply@berkeley...|160000|
| michael| jackson| no-reply@neverla.nd| 80000|
| null| wendell|no-reply@berkeley...|160000|
| xiangrui| meng|no-reply@stanford...|120000|
| matei| null|no-reply@waterloo...|140000|
+---------+--------+--------------------+------+
ahora cuando pruebo con otra columna me pasa esto
from pyspark.sql.functions import explode
explodeDF = comoquieradf.select(explode("department").alias("e"))
flattenDF = explodeDF.selectExpr("e.id", "e.name")
flattenDF.show()
output
---------------------------------------------------------------------------
AnalysisException Traceback (most recent call last)
<ipython-input-37-e8603541b35a> in <module>
1 from pyspark.sql.functions import explode
2
----> 3 explodeDF = comoquieradf.select(explode("department").alias("e"))
4 flattenDF = explodeDF.selectExpr("e.id", "e.name")
5
2 frames
/usr/local/lib/python3.7/dist-packages/pyspark/sql/utils.py in deco(*a, **kw)
194 # Hide where the exception came from that shows a non-Pythonic
195 # JVM exception message.
--> 196 raise converted from None
197 else:
198 raise
AnalysisException: cannot resolve 'explode(department)' due to data type mismatch: input to function explode should be array or map type, not struct<id:string,name:string>;
'Project [explode(department#0) AS e#82]
+- Relation [department#0,employees#1] parquet
Me podran ayudar??