I am working on some migration project where we need to migrate data from legacy system to new system. Each system has its own schema and hence we need transformations as well. Say for example in legacy we have date in yyyy-mm-dd format where as in new we have epoch, same way Legacy system code to New system code etc.,.
One notable things here is 200M records really does not stand on 1 relations, those are 200M logical business object. So each logic business object intern composes of many tables (roughly I could say 10-12 relations with average of 50 records all together for each logical business object)
In our earlier migration we have a spring batch job (multithreaded with 4 threads) running on company infrastructure with 16GB RAM and 4 core CPU taking 5 logical business objects to migrate in 2 second. Obviously we cannot go with this for this bulk migration. Looking for alternative like using Parallel execution in PL/SQL procedure or any other means.
We cannot rely on ETL team as time lines does not match with our project deadlines. Is there any efficient way to handle this?
Jagan