in hql scripts we use "!sh echo ---new line---" for the same . Want to know the alternative for this in impala to print any line in impala scripts?
in hql scripts we use "!sh echo ---new line---" for the same . Want to know the alternative for this in impala to print any line in impala scripts?
You can invoke the shell command line from impala scripts.
As an example of how it works.
script_impala.sql
-- set a variable containing the of the game
SET hivevar:game=Monopoly;
-- return the list of the game
SELECT list_price FROM fun.games WHERE name = '${hivevar:game}';
-- return the prices of the game ate game shops
SELECT shop, price FROM fun.inventory WHERE game = '${hivevar:game}';
shell hdfs dfs -ls /user;
shell ls -ltr;
shell echo I can echo from impala scripts;
shell cat hex_color.sql
$ impala-shell -f script_impala.sql
output
Starting Impala Shell without Kerberos authentication
Connected to localhost.localdomain:21000
Server version: impalad version 2.10.0-cdh5.13.3 RELEASE (build 15a453e15865344e75ce0fc6c4c760696d50f626)
Variable GAME set to Monopoly
Query: -- return the list of the game
SELECT list_price FROM fun.games WHERE name = 'Monopoly'
Query submitted at: 2020-06-04 23:29:19 (Coordinator: http://localhost.localdomain:25000)
Query progress can be monitored at: http://localhost.localdomain:25000/query_plan?query_id=a94afc7556843cf7:8b19809d00000000
+------------+
| list_price |
+------------+
| 19.99 |
+------------+
Fetched 1 row(s) in 0.15s
Query: -- return the prices of the game ate game shops
SELECT shop, price FROM fun.inventory WHERE game = 'Monopoly'
Query submitted at: 2020-06-04 23:29:19 (Coordinator: http://localhost.localdomain:25000)
Query progress can be monitored at: http://localhost.localdomain:25000/query_plan?query_id=c64d1c8950e5e1f8:15d410cb00000000
+-----------+-------+
| shop | price |
+-----------+-------+
| Dicey | 17.99 |
| Board 'Em | 25.00 |
+-----------+-------+
Fetched 2 row(s) in 0.14s
Found 5 items
drwxrwxrwt - mapred hadoop 0 2019-04-29 18:19 /user/history
drwxrwxrwx - hive hive 0 2019-04-29 18:19 /user/hive
drwxr-xr-x - hue hue 0 2019-11-25 10:19 /user/hue
drwxr-xr-x - spark spark 0 2019-04-29 18:19 /user/spark
drwxrwxrwx - training supergroup 0 2020-05-28 11:33 /user/training
--------
Executed in 2.10s
total 56
-rw-rw-r-- 1 training training 61 Sep 25 2019 hex_color.sql
-rw-rw-r-- 1 training training 115 Sep 25 2019 color_from_rgb.sql~
-rw-rw-r-- 1 training training 58 Sep 25 2019 hex_color_impala.sql
-rwxr-xr-x 1 training training 449 Sep 25 2019 email_results.sh
-rw-rw-r-- 1 training training 1166 Sep 25 2019 zero_air_time.csv
-rw-r--r-- 1 training training 261 Sep 26 2019 change_background.sh~
-rwxr-xr-x 1 training training 262 Sep 26 2019 change_background.sh
-rw------- 1 training training 2966 Sep 26 2019 ChangeVMDesktopColor.txt
-rw------- 1 training training 3279 Sep 26 2019 Hive&ImpalaInScripts&Applications.txt~
-rw------- 1 training training 3278 Sep 26 2019 Hive&ImpalaInScripts&Applications.txt
-rw-rw-r-- 1 training training 449 Sep 26 2019 email_resuts.sh
-rw-rw-r-- 1 training training 120 Apr 27 11:38 color_from_rgb.sql
-rw-rw-r-- 1 training training 397 Jun 4 23:24 game_prices.sql~
-rw-rw-r-- 1 training training 395 Jun 4 23:29 game_prices.sql
--------
Executed in 0.00s
I can echo from impala scripts
--------
Executed in 0.00s
SELECT hex FROM wax.crayons WHERE color = '${hivevar:color}'
--------
Executed in 0.01s
As you can see, you only have to write in your script
shell <command>;
You can even invoke hdfs dfs commands
shell hdfs dfs -<command>;
The above example prints the output to the console, if you want to print the output to a file you have to do as following to redirect all the output to a file:
$ impala-shell -f script_impala.sql >> /home/..../my_file.csv
Other option would be to redirect only the echo commands from inside the script. This would be as following: script_impala.sql
-- set a variable containing the of the game
SET hivevar:game=Monopoly;
-- return the list of the game
SELECT list_price FROM fun.games WHERE name = '${hivevar:game}';
-- return the prices of the game ate game shops
SELECT shop, price FROM fun.inventory WHERE game = '${hivevar:game}';
shell hdfs dfs -ls /user;
shell ls -ltr;
shell echo I can echo from impala scripts >> /home/..../my_file.csv;
shell echo How are you? >> /home/..../my_file.csv;
shell echo I am feeling very good, how about you? >> /home/..../my_file.csv;
shell cat hex_color.sql
and don't forget the semicolon at the end of the line ";". I hope it helps.