Beeline getting stuck on the second command when run through GNU parallel

user_146625 0
Viewed 257

We are trying to run a fixed number of Hive queries in parallel using GNU parallel. Even with parallelism set to 1 (i.e. sequential execution) via -j1 the first execution works, but the second one gets stuck:

$ parallel -j1 --eta --verbose beeline -e '"SELECT \"{}\";"' ::: a b c
beeline -e "SELECT \"a\";"

Computers / CPU cores / Max jobs to run
1:local / 40 / 1

Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
ETA: 0s Left: 3 AVG: 0.00s  local:1/0/100%/0.0s
+------+
| _c0  |
+------+
| a    |
+------+
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH/jars/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'log4j2.debug' to show Log4j2 internal initialization logging.
WARNING: Use "yarn jar" to launch YARN applications.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH/jars/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Connecting to jdbc:hive2://en02.example.cloud:2181,mn01.example.cloud:2181,mn02.example.cloud:2181/default;principal=hive/_HOST@example.cloud;serviceDiscoveryMode=zooKeeper;ssl=true;zooKeeperNamespace=hiveserver2
21/11/16 10:41:39 [main]: INFO jdbc.HiveConnection: Connected to mn01.example.cloud:10000
Connected to: Apache Hive (version 3.1.3000.7.1.6.0-297)
Driver: Hive JDBC (version 3.1.3000.7.1.6.0-297)
Transaction isolation: TRANSACTION_REPEATABLE_READ
INFO  : Compiling command(queryId=hive_20211116104139_7be8d8ee-f58d-4572-88a4-43533846160b): SELECT "a"
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:string, comment:null)], properties:null)
INFO  : Completed compiling command(queryId=hive_20211116104139_7be8d8ee-f58d-4572-88a4-43533846160b); Time taken: 0.102 seconds
INFO  : Executing command(queryId=hive_20211116104139_7be8d8ee-f58d-4572-88a4-43533846160b): SELECT "a"
INFO  : Completed executing command(queryId=hive_20211116104139_7be8d8ee-f58d-4572-88a4-43533846160b); Time taken: 0.006 seconds
INFO  : OK
1 row selected (0.191 seconds)
Beeline version 3.1.3000.7.1.6.0-297 by Apache Hive
Closing: 0: jdbc:hive2://en02.example.cloud:2181,mn01.example.cloud:2181,mn02.example.cloud:2181/default;principal=hive/_HOST@example.cloud;serviceDiscoveryMode=zooKeeper;ssl=true;zooKeeperNamespace=hiveserver2
beeline -e "SELECT \"b\";"
ETA: 79s Left: 2 AVG: 42.00s  local:1/1/100%/47.0s

Simplifying this further, even a parallel call to beeline --help gets stuck for the second run in the same way, so it doesn't seem to be related to the connection to the Hive DB.

gnu-parallel hive stdin freeze
1 Answers

The solutions with which we finally got it working is

parallel -j1 --eta --verbose beeline -e '"SELECT \"{}\";"' < /dev/null ::: a b c

and (thanks @OleTange !)

parallel -j1 --eta --verbose --tty beeline -e '"SELECT \"{}\";"' ::: a b c

How we found out: We added a set -x to the beeline bash script and some of the scripts it's calling, logged the results to separate files for the parallel runs and diffed them. We saw that there was a part in the logs about

[ -p /dev/stdin ]

and below that a few environment variables that got set in the first parallel execution, but not on the second one. We then played around with various options to give beeline a stdin and the /dev/null version finally worked.

user_1466250
Related