I am busy upgrading our AWS EMR cluster to a newer version and I have started running into issues.
The errors are only occur on the newer version (Hive 3.1.3).
All my queries run fine on the older version (Hive 2.8.5)
The error I get when trying to insert into or insert overwrite is:
Could not connect to any of [('IP', port)] (code THRIFTTRANSPORT): TTransportException("Could not connect to any of [('IP', port)]",)
AND
TSocket read 0 bytes
I use Hive to do some ETL.
We host an EMR cluster that uses Hive, Hue, Tez, Pig.
The tables are build as external tables on parquet files stored in S3.
Some of these tables are partitioned.
On the current version, these are the sets I use before running the queries:
SET hive.vectorized.execution.enabled = true;
SET hive.vectorized.execution.reduce.enabled = true;
SET hive.vectorized.execution.reduce.groupby.enabled = true;
SET hive.cbo.enable=true;
SET hive.compute.query.using.stats=true;
SET hive.stats.fetch.column.stats=true;
SET hive.stats.fetch.partition.stats=true;
SET hive.execution.engine=mr;
SET hive.exec.parallel=true;
SET hive.exec.dynamic.partition.mode=nonstrict;
SET hive.exec.max.dynamic.partitions=10000;
SET hive.exec.max.dynamic.partitions.pernode=10000;
SET hive.exec.max.created.files=500000;
SET hive.metastore.sasl.enabled = true; --Newly add this but still getting these errors
On the later version this set no longer works:
SET hive.stats.fetch.partition.stats=true;
Current Version:
Release label:emr-5.27.0
Hadoop distribution:Amazon 2.8.5
Applications:Hive 2.3.5, Hue 4.4.0, Tez 0.9.2, Pig 0.17.0, Hadoop 2.8.5
Wanted version:
Release label:emr-6.7.0
Hadoop distribution:Amazon 3.2.1
Applications:Hive 3.1.3, Hue 4.10.0, Tez 0.9.2, Pig 0.17.0, Hadoop 3.2.1
Are there SET's that I am missing or is there something else I am missing?
Any advise would be greatly appreciated.