Check the Schema of all ingested files

Viewed 48

I have a dataset DF1 which is ingested from an external data source into foundry and it is stored as .parq files in foundry.

My aim to identify files that may have different schema from others. In other words I want to

  1. create a list of all parquet files,
  2. iterate over that list so that for each ingested file: (a) read the schema and (b) record the results in a new data frame DF2 with columns: the path and the number of columns for each ingested file.
  3. output the result as a dataframe DF2.

Thanks in advance.

0 Answers
Related