I have the following dataframe ( shown below) where i have an array of struct. I am trying to compare the values within the struct to a black list and eliminate all matches from the array. I came across multiple posts with similar situations but i am held up at how to pass a struct to my UDF that will iterate through the black list and find matches.
Example , blacklist: List[String] : {'Tom',Harry'}
so authors matching the two names if found will be removed from the struct schema Unable to post image due to being a new user
Dataframe
val arrayStructSchema = new StructType().add("name",StringType)
.add("booksIntersted",ArrayType(new StructType()
.add("name",StringType)
.add("author",StringType)
.add("pages",IntegerType)))
val df = spark.createDataFrame(
spark.sparkContext.parallelize(arrayStructData),arrayStructSchema)
df.printSchema()
df.show(false)
Udf to match string in blacklist to column booksInterested
def filterBooks(books: List[String]) = udf {
(recs: List[String]) => recs.filter(blacklist =>books.contains(blacklist))
}
I get a type error since i am unable to iterate the struct