Shell script for loading hive table into Apache solr is failing

user_10837808 0
Viewed 160

I am writing a shell script to load data from a hive table into Apache solr schema.

SRC=$1
    
TGT=$2
    
SOLR_COLLECTION=$3
    
SCHEMA=$(python schemaFetch.py ${SRC})
    
hive -e "ADD JAR /hadoopfs/fs1/dev/solr-hive-serde-3.0.0.jar;
    create external table if not exists ${TGT}${SCHEMA}
    row format delimited
    fields terminated by ','
    STORED BY 'com.lucidworks.hadoop.hive.LWStorageHandler'
    TBLPROPERTIES('solr.server.url' = 'http://10.165.137.139:8886/solr/',
    'solr.collection' = '${SOLR_COLLECTION}',
    'solr.query' = '*:*');"

hive -e "insert into ${TGT} select * from ${SRC};"

But it is showing a runtime exception:

FAILED: RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.com.lucidworks.hadoop.hive.LWStorageHandler

Yet also schema is being created in particular hive location and below listed insert into query is not working from the script.

hive -e "insert into ${TGT} select * from ${SRC};

I do not get what is happening.

hive solr sh python
1 Answers

The command hive -e "insert into ${TGT} select * from ${SRC}; is missing a ".

Assuming it should be hive -e "insert into ${TGT} select * from ${SRC};", you may be running into a problem with shell globbing: the * is being replaced with the contents of the folder from which you are running the command.

Consider running the command either as hive -e "insert into ${TGT} select * from ${SRC};" or by doing this (using bash syntax):

set -o noglob
hive -e "insert into ${TGT} select * from ${SRC};"
set +o noglob

The first command disables globbing, so that * is not interpreted, while the third restores it.

You can use a similar technique in your script to ensure * isn't interpreted inappropriately.

user_105594870
Related