Spark scala join with duplicate columns when used finial result df.*

Viewed 21

When I join two dataframes using Seq(col), I still get multiple columns when using df.*

E.g: '''

val schema = StructType( Array(
                 StructField("language", StringType,true),
                 StructField("users", StringType,true)
             ))
val rowData= Seq(Row("Java", "20000"), 
               Row("Python", "100000"), 
               Row("Scala", "3000"))

var dfFromData3 = spark.createDataFrame(rowData,schema)

val schema1 = StructType( Array(
                 StructField("language", StringType,true),
                 StructField("price", StringType,true)
             ))
val rowData1= Seq(Row("Java", "20"), 
               Row("Python", "10")) 
              
var dfFromData4 = spark.createDataFrame(rowData1,schema1)

val combined = dfFromData3.join(dfFromData4,Seq("language"),"left")

''' '''display(combined)''' - Has only one "language" column

but

'''display(combined.as("df").select("df.*"))''' - Has two "language" columns

Can someone please explain what is happening here?

0 Answers
Related