We use the distcp tool to copy data from one cluster(C1) to another cluster(C2) with TB of data everyday.
C1 directory structure:
"/user/ten/warehouse/tmp.db/na2_rasal/dt=2022-06-30/000000_0"
At present, we use following command which successful in copying the data.
"--delete --update /user/ten/warehouse/tmp.db/na2_rasal /user/ten/warehouse/tmp.db/na2_rasal"
After copying, we modify the table(na2_rasal) in C2 to convert to an orc table using CTAS. Now, the table in C2 is ORC data.
The issue happens when we want to copy again for the same table, because C1 table is sequential format whereas C2 table is ORC. In this case, we convert back the C2 table back to sequential, recopy and again convert to ORC. This becomes a tedious task.
I wanted to try maintaining an intermediate table in C2 which is the same format as the C1 table. Hence want to copy something like,
trial 1:
"/user/ten/warehouse/tmp.db/na2_rasal/ /user/ten/warehouse/tmp.db/na2_rasal_sequence"
result in C2 directory:
/user/ten/warehouse/tmp.db/na2_rasal_sequence/na2_rasal/dt=2022-06-30/000000_0 -----this does not work with difference name, structure gets changed here.
trial 2:
"/user/ten/warehouse/tmp.db/na2_rasal/* /user/ten/warehouse/tmp.db/na2_rasal_sequence"
result in C2 directory:
/user/ten/warehouse/tmp.db/na2_rasal_sequence/dt=2022-06-30/000000_0 --this works fine, but cannot use -update/overwrite since use of recursive source directory
By this method, I can copy the data from na2_rasal from C1 to na2_rasal_sequence in C2. But I am afraid, how feasible is this solution for TB of data?
Do you have any other solution? Please advise me. Thank you.