How to add new field to struct column?

user_1392008 0
Viewed 12200

I have a dataframe with something like this df.printSchema:

root
|-- ts: timestamp (nullable = true)
|-- geoip: struct (nullable = true)
|    |-- city: string (nullable = true)
|    |-- continent: string (nullable = true)
|    |-- location: struct (nullable = true)
|    |    |-- lat: float (nullable = true)
|    |    |-- lon: float (nullable = true)

I know that for example with df = df.withColumn("error", lit(null).cast(StringType)) I can add a null field called error of type string right under the root. How could I add the same field under the geoip struct, or under the location struct?

I have also tried df = df.withColumn("geoip.error", lit(null).cast(StringType)) with no luck.

apache-spark-sql apache-spark field struct scala
4 Answers

You could also simply do:

df = df.withColumn("goip", struct($"geoip.*", lit("This is fine.").alias("error")))

That adds an "error" field to the "geoip" struct.

user_17716340

Spark 3.1+

col("geoip").withField("error", lit(null).cast("string"))

Example input:

val df = Seq(("Vilnius", "Europe", 1, 1)).toDF("city", "continent", "lat", "lon")
         .withColumn("location", struct("lat", "lon").as("location"))
         .select(struct("city", "continent", "location").as("geoip"))
df.printSchema()
// root
//  |-- geoip: struct (nullable = false)
//  |    |-- city: string (nullable = true)
//  |    |-- continent: string (nullable = true)
//  |    |-- location: struct (nullable = false)
//  |    |    |-- lat: integer (nullable = false)
//  |    |    |-- lon: integer (nullable = false)

Example #1

val df2 = df.withColumn("geoip", col("geoip").withField("error", lit(null).cast("string")))
df2.printSchema()
// root
//  |-- geoip: struct (nullable = false)
//  |    |-- city: string (nullable = true)
//  |    |-- continent: string (nullable = true)
//  |    |-- location: struct (nullable = false)
//  |    |    |-- lat: integer (nullable = false)
//  |    |    |-- lon: integer (nullable = false)
//  |    |-- error: string (nullable = true)

Example #2

val df3 = df2.withColumn("geoip", col("geoip").withField("location.error", lit(null).cast("string")))
df3.printSchema()
// root
//  |-- geoip: struct (nullable = false)
//  |    |-- city: string (nullable = true)
//  |    |-- continent: string (nullable = true)
//  |    |-- location: struct (nullable = false)
//  |    |    |-- lat: integer (nullable = false)
//  |    |    |-- lon: integer (nullable = false)
//  |    |    |-- error: string (nullable = true)
//  |    |-- error: string (nullable = true)
user_27535010
Related