How to store the result of Google Dataproc query inside a variable GCP

user_11277781 0
Viewed 489

I have a requirement wherein I need to count the number of records in a gcloud hive table and need to store this result inside a variable.

Below is the code for the same:

test=$(gcloud dataproc jobs submit hive --cluster=$CLUSTER --region=$REGION --execute="select count(*) from db.table;")

However, the above variable is not storing the count of records but is storing some logs which not useful for me.

Can please someone help me to find out how can we redirect the output of above query inside a variable.

google-cloud-dataproc google-cloud-platform
2 Answers

My guess is the output you mention includes the logs for the Hive command and the output you want. It sounds like you just want the latter.

I'd recommend using something like grep, sed, or Python to capture the output. If you know regular expressions (regex), this should be pretty easy - this is a good example of what you might want to do. If you have not used regex before, a regex builder like this one will be useful.

user_46447430

The output of gcloud command actually consists of two parts: stderr and standard output. The output that includes count number is actually insided the stderr. The following command can do this trick,

cnt_output=$((gcloud dataproc jobs submit hive --cluster=$CLUSTER --region=$REGION --execute "select count(*) from db.table;" 1>/dev/null) 2>&1)

This basically strips the standout first and then convert the stderr to standard output so that it can be saved into a variable, i.e. cnt_output

After that you can use the tool mentioned in the answer above to capture the number you want.

user_127347460
Related