Filter an array Struct column based on a provided list

Viewed 51

I have the following dataframe ( shown below) where i have an array of struct. I am trying to compare the values within the struct to a black list and eliminate all matches from the array. I came across multiple posts with similar situations but i am held up at how to pass a struct to my UDF that will iterate through the black list and find matches.

Example , blacklist: List[String] : {'Tom',Harry'}

so authors matching the two names if found will be removed from the struct schema Unable to post image due to being a new user

Dataframe

 val arrayStructSchema = new StructType().add("name",StringType)
    .add("booksIntersted",ArrayType(new StructType()
      .add("name",StringType)
      .add("author",StringType)
      .add("pages",IntegerType)))

  val df = spark.createDataFrame(
    spark.sparkContext.parallelize(arrayStructData),arrayStructSchema)
  df.printSchema()
  df.show(false)

Udf to match string in blacklist to column booksInterested

def filterBooks(books: List[String]) = udf {
  (recs: List[String]) => recs.filter(blacklist =>books.contains(blacklist))
}

I get a type error since i am unable to iterate the struct

0 Answers
Related