extract elements in dataframe row

user_11469656 0
Viewed 315

I struggle accessing data in my Dataframe. More specifically two particular columns (back&front) where the structure varies from row to row. I would like to extract the string value of the fields called text

df.select(df['back'])
DataFrame[back: struct<attrs:struct<version:int>,content:array<struct<content:array<struct<content:array<struct<type:string,content:array<struct<type:string,text:string>>>>,text:string,type:string>>,type:string>>,type:string>]

df.select(df['back']).show()
|[[1], [[[[, Vernetzte Komponenten innerhalb eines Netzwerks mit verschiedenen Hardware-Rechnern realisiert Ja., text]], paragraph]], doc]    

df.select(df['back']).collect()
Row(back=Row(attrs=Row(version=1), content=[Row(content=[Row(content=None, text='Vernetzte Komponenten innerhalb eines Netzwerks mit verschiedenen Hardware-Rechnern realisiert Ja.', type='text')], type='paragraph')], type='doc'))
pyspark apache-spark mongodb dataframe python
1 Answers

This function iterates over your nested column and it retrieves all text values:

import pyspark.sql.functions as f

@f.udf()
def extract_texts(elements):
  texts = []
  
  while len(elements) > 0:
    element = elements.pop(0)
    if 'content' in element and element['content'] is not None:
      elements.extend(element['content'])
      
    if 'text' in element:
      texts.append(element['text'])

  return texts

new_df = (df
          .withColumn('back_texts', extract_texts('back.content'))
          .withColumn('front_texts', extract_texts('front.content')))

new_df.show()
user_60802760
Related