Best Practice for processing unstructured data with Apache Beam

user_9383039 0
Viewed 180

We're using Airflow for job scheduling, and calling Apache Beam for the ETL step. The data source is unstructured files (batch) which need to be parsed before they can be turned into PCollections. It appears to me that the two best options available are:

  1. Add a preprocessing node to the Airflow DAG to parse the files and write to a parquet file, which is then processed by Beam.
  2. Write a custom IO connector in Beam to parse the unstructured file and create the PCollection.

Which option better fits Beam best practices?

apache-beam airflow etl architecture python
2 Answers

I'd vote for 1) if you need to re-use these files for other pipelines later and parsing these unstructured files takes significant amount of time.

On the other hand, if parsing these files can be run in parallel and you don't need to wait for all files to be ready, then I'd choose 2).

Anyway, I think it will be a trade-off that depends on your needs and input data.

user_106873250

In my opinion the most important part of an ETL is not what does when it works perfectly; but how you handle the rejects (errors, incomplete data, etc).

If you can reuse the code, then #1 works, but my bet is #2 because all the code that handles the ETL lies together.

If you don't want to write a customIO but wanted to execute some external application to parse the data, you can use a custom docker container for a dataflow job.

user_67603720
Related