Spark: Read multiple AVRO files with different schema in parallel

Viewed 862

I have many (relatively small) AVRO files with different schema, each in one location like this:

Object Name: A
/path/to/A
   A_1.avro
   A_2.avro
   ...
   A_N.avro

Object Name: B
/path/to/B
   B_1.avro
   B_2.avro
   ...
   B_N.avro

Object Name: C
/path/to/C
   C_1.avro
   C_2.avro
   ...
   C_N.avro
...

and my goal is to read them in parallel via Spark and store each row as a blob in one column of the output. As a result my output data will have a consistent schema, something like the following columns: ID, objectName, RecordDate, Data Where the 'Data' field contains a string JSON of the original record.

My initial thought was to put the spark read statements in a loop, create the fields shown above for each dataframe, and then apply a union operation to get my final dataframe, like this:

all_df = []
for obj_name in all_object_names:
    file_path = get_file_path(object_name)
    df = spark.read.format(DATABRIKS_FORMAT).load(file_path)
    all_df.append(df)

df_o = all_df.drop()
for df in all_df:
    df_o = df_o.union(df)

# write df_o to the output

However I'm not sure if the read operations are going to be parallelized.

I also came across the sc.textFile() function to read all the AVRO files in one shot as string, but couldn't make it work.

So I have 2 questions:

  1. Would the multiple read statements in a loop be parallelized by Spark? Or is there a more efficient way to achieve this?
  2. Can sc.textFile() be used to read the AVRO files as a string JSON in one column?

I'd appreciate your thoughts and suggestions.

0 Answers
Related