I have a table in which one field is having json data (around 135 fields in json format). now i have a script which is selecting this json data and loading this 135 fields into another table (table 2) having exact 135 similar columns. Below is the sample format of fields in table 1 and respective columns in table 2.
table1
column1
{"transactiondetails":{"transaction_id":"123","timestamp":"2020-01-01T09:03:57" ---____-----____----____},
"inquiry_details":{inquiry_id":"123","Language":"English(us)","postal_code":"123456" -----_____-----_____},
128 more fields similar to this....
table2- columns name
transactionid, timestamp, inquiry_id, language, postal_code..... 128 more columns
the issue with the hql is it's taking a lot of time to run.The hql script is a simple insert overwrite into table2 by selecting data from table 1. Below is the performance tuning parameters i am using .
set.spark.executor.extrajavaOptions=-Xss16m
set hive.execution.engine=spark;
set mapreduce.map.memory.mb=4096;
set spark.master=yarn-cluster;
set spark.serializer=org.apache.spark.serializer.kyroserializer;
set hive.exec.dynamic.partitioning=true;
set hive.exec.dynamic.partitioning.mode=nonstrict;
set hive.execute.parallel=true;
set hive.exec.parallel.thread.number=10;
is there any way to reduce the time taken to do this task.