aws glue maches dynamic record

Question

Could anyone explain code below for matching record in Aws glue from manual https://aws.amazon.com/ru/blogs/big-data/work-with-partitioned-data-in-aws-glue/

I printed the full schema using githubEvents.printSchema():

id: string
type: string
actor: struct
repo: struct
payload: struct
public: boolean
created_at: string
year: string
month: string
day: string
org: struct

def filterWeekend(rec: DynamicRecord): Boolean = {
  def getAsInt(field: String): Int = {
    rec.**getField(field**) match {
      case Some(strVal: String) => strVal.toInt
      // The filter transformation will catch exceptions and mark the record as an error.
      case _ => throw new IllegalArgumentException(s"Unable to extract field $field")
    }
  }

  val (year, month, day) = (getAsInt("year"), getAsInt("month"), getAsInt("day"))
  val cal = new GregorianCalendar(year, month - 1, day) // Calendar months start at 0.
  val dayOfWeek = cal.get(Calendar.DAY_OF_WEEK)

  **dayOfWeek** == Calendar.SATURDAY || dayOfWeek == Calendar.SUNDAY
}

val filteredEvents = githubEvents.filter(filterWeekend)
filteredEvents.count

I think that day should be ingested in getField(field) how glue knows that day should be compared but not other field so I mean we should compare day of the schema with dayofWeek?

aws glue maches dynamic record

0 Answers