Approach or tools to transform raw data into usable formats - Data Consolidation, Data Preparation

Sakthivel 0
Viewed 19

We are working on data preparation activity for a Datawarehousing project to build an enterprise level data warehouse. As an intermediate layer, we are bringing all the source database information into Hadoop datalake as hive tables.

In our use case, we have around 3000+ source tables. We would like to consolidate the source data into a new schema in bigdata datalake. As part of this, we need to do the data profiling / data preparation for all the source database tables by analysing each and every columns to map/group into the respective meaningful destination table.

Requesting suggesting from the experts on any tools (open source or licensed) to easy the process.

Thanks, Sakthivel

data-wrangling data-preprocessing data-warehouse
1 Answers

Here are a few questions to better understand the use case -

  1. Does the preparation requires joins?
  2. SQL / NoSQL?
  3. Cloud (Which one?) / On-prem?
  4. Can the process happen in real-time?
Yaniv Ben Hemo0
Related