Parse Json file in HDFS and write to output only if key is available

Padfoot123 0
Viewed 30

I have a JSON file with "name" as key and "url" as another key. Not all records in the JSON have the "url" key. So when I am trying to write to text it gets written as null. So, I will have to do a check if url key is present and if present, then it has to be written to 2 output text files. name will goto name.txt and url will goto url.txt

{ [ {"name": "a", url: "url1"}, {"name": "b"}, {"name": "C"} ] }

Expected output:

name.txt
a
url.txt
url

What i tried so far:

name=`hdfs dfs -cat $path | jq -r '.name'`
url=`hdfs dfs -cat $path | jq -r '.name'`

echo ${name} >> name.txt
echo ${url} >> url.txt

The problem with above is if url is not found it writes a null to output file and equivalent name is written to name.txt file.

How to skip the records from writing to output file when url json tag is not found

hdfs jq shell
1 Answers

Evidently you need first to select(.url).

Calling both hdfs and jq twice seems a bit extravagant. If efficiency is a concern at all, you might wish to save the result of hdfs to a file and call jq twice, or perhaps even better to contrive to call hdfs and jq just once in a pipeline that ends with a call to awk to write the two files.

peak0
Related