How to create DF with exclude of just some fields from ready schema

QbS 0
Viewed 18

I have a schema:

schema = StructType(
    [
        StructField("COUNTRY", StringType(), False),
        StructField("ID", StringType(), False),
        StructField("DATE", DateType(), False),
        ... x80
]

and I'd like to create a DF with only 30 of fields from such schema. How to do it in an efficient way?

What I do, but it seems not eficient - I'm looking for more effective way. Is there such?

created_df_with_only_few_fields_from_the_schema = self.create_df(
                    [
                     {
                      "COUNTRY": "Germany",
                      ... x30
                    },schema["ID","DATE",... x30] --> all the fields that I'd like to extract
                     ]
schema pyspark
1 Answers

So you can just read the data with all the fields first, and then select the ones you need. e.g.,

selected_cols = ["col1", "col3", "col10", ... 30 fields]
selected_df = original_df[selected_cols]
the_ordinary_guy0
Related