How to create a list or dataframe

Viewed 39

I'm testing a case where whatever columns are included in Join are actually present in the correct dataframe or not.

Let's say i have 2 DF's defined as a case class

case class JoinResult(
    id: Option[String],
    name: Option[String],
    maxAttack: Option[Int],
    maxDefense: Option[Int],
    cartoon: Option[String]
)

case class JoinSchema(
    uuid: String,
    cartoon: String
)

In this JoinResult schema is the parentDF and JoinSchema is childDF

To join this i'm running a spark job which will take a dslString Something like

"""
        |{
        |  "with": "D2",
        |  "type": "left",
        |  "keys": {
        |             "D1.id": "D2.uuid"
        |           },
        |  "colList": [
        |                  {
        |                     "col": "D1.id",
        |                     "as": "id"
        |                  },
        |                  {
        |                     "col": "D1.name",
        |                     "as": "name"
        |                  },
        |                  {
        |                     "col": "D1.max_attack",
        |                     "as": "max_attack"
        |                  },
        |                  {
        |                     "col": "D1.max_defense",
        |                     "as": "max_defense"
        |                  },
        |                  {
        |                     "col": "D2.cartoon",
        |                     "as": "cartoon"
        |                  },
        |                  {
        |                     "col": "D2.test_cols",
        |                     "as": "test_cols"
        |                  }
        |              ]
        |}
        |""".stripMargin

My End requirement is to check all the cols which start with D2 aliasing should be there in childDF and cols starting with D1 aliasing should be there in parentDF And if none of the column is there then it should throw me an error.

To achieve this i'm writing a function to iterate over the col present in the dslString and the ones starting with D1 should be stored into parentInvCols and cols starting with D2 should be stored in childInvCols.

Here is the main function

config.colList.foreach { cInfo =>
      val colInfo = cInfo.col.split("\\.")
      if (childAlias.equals(colInfo.head)) {
        colInfo.last
      }
    }

So as per the code if childAlias = colInfo.head means D2 = D2 then it should store in childInvCols

1 Answers

Something like this perhaps?

val cols: Map[String, Seq[String]] = config
   .colList
   .map(_.col)
   .map(_.split("\\."))
   .groupMap(_.head)(_.last)

It creates a map with prefixes (D1, D2 etc.) as keys and lists of corresponding column names as values

Related