One can access PySpark Row elements using the dot notation: given r= Row(name="Alice", age=11), one can get the name or the age using r.name or r.age respectively. What happens when one needs to get an element whose name is stored in a variable element? One option is to do r.toDict()[element]. However, consider a situation where we have a large DataFrame and we wish to map a function on each row of that data frame. We can certainly do something like
def f(row, element1, element2):
row = row.asDict()
return ", ".join(str(row[element1]), str(row[element2]))
result = dataframe.map(lambda row: f(row, 'age', 'name'))
However, it seems that calling toDict() on every row will be very inefficient. Is there a better way?