How to get values in a line while looping line by line in a file (shell script)

user_16208559 0
Viewed 72

I have a file which looks like this (file.txt)

{"key":"AJGUIGIDH568","rule":squid:111-some_random_text_here
{"key":"TJHJHJHDH568","rule":squid:111-some_random_text_here
{"key":"YUUUIGIDH566","rule":squid:111-some_random_text_here
{"key":"HJHHIGIDH568","rule":squid:111-some_random_text_here
{"key":"ATYUGUIDH556","rule":squid:111-some_random_text_here
{"key":"QfgUIGIDH568","rule":squid:111-some_random_text_here

I want to loop trough this line by line an extract the key values.

so the result should be like ,

AJGUIGIDH568
AJGUIGIDH568
YUUUIGIDH566
HJHHIGIDH568
ATYUGUIDH556
QfgUIGIDH568

So I wrote a code like this to loop line by line and extract the value between {"key":" and ","rule": because key values is in between these 2 patterns.

while read p; do 
  echo $p | sed -n "/{"key":"/,/","rule":,/p"
done < file.txt

But this is not working. can someone help me to figure out me this. Thanks in advance.

sed grep awk shell
5 Answers

For the shown data, you can try this awk:

awk -F '"[:,]"' '{print $2}' file

AJGUIGIDH568
TJHJHJHDH568
YUUUIGIDH566
HJHHIGIDH568
ATYUGUIDH556
QfgUIGIDH568
user_5482250

With the give example you can simple use

cut -d'"' -f4 file.txt
user_32201130

Your sample input is almost valid json. You could tweak it to make it valid and then extract the values with jq with something like:

sed -e 's/squid/"squid/' -e 's/$/"}/' file.txt | jq -r .key

Or, if your actual input really is valid json, then just use jq:

jq -r .key file.txt

If the "random-txt" may include double quotes, making it difficult to massage the input to make it valid json, perhaps you want something like:

awk '{print $4}' FS='"' file.txt

or

sed -n '/{"key":"\([^"]*\).*/s//\1/p' file.txt

or

while IFS=\" read open_brace key colon val _; do echo "$val"; done < file.txt
user_1407500

Using sed

$ sed -E 's/([^"]*"){3}([^"]*).*/\2/' input_file
AJGUIGIDH568
TJHJHJHDH568
YUUUIGIDH566
HJHHIGIDH568
ATYUGUIDH556
QfgUIGIDH568
user_163721090

Assumptions:

  • there may be other lines in the file so we need to focus on just the lines with "key" and "rule"
  • the only text between "key" and "rule" is the desired string (eg, squid never shows up between the two patterns of interest)

Adding some additional lines:

$ cat file.txt
{"key":"AJGUIGIDH568","rule":squid:111-some_random_text_here
ignore this line}
{"key":"TJHJHJHDH568","rule":squid:111-some_random_text_here
ignore this line}
{"key":"YUUUIGIDH566","rule":squid:111-some_random_text_here
ignore this line}
{"key":"HJHHIGIDH568","rule":squid:111-some_random_text_here
ignore this line}
{"key":"ATYUGUIDH556","rule":squid:111-some_random_text_here
ignore this line}
{"key":"QfgUIGIDH568","rule":squid:111-some_random_text_here
ignore this line}

One sed idea:

$ sed -nE 's/^(.*"key":")([^"]*)(","rule".*)$/\2/p' file.txt
AJGUIGIDH568
TJHJHJHDH568
YUUUIGIDH566
HJHHIGIDH568
ATYUGUIDH556
QfgUIGIDH568

Where:

  • -E - enable extended regex support (and capture groups without need to escape sequences)
  • -n - suppress printing of pattern space
  • ^(.*"key":") - [1st capture group] everything from start of line up to and including "key":"
  • ([^"]*) - [2nd capture group] everything that is not a double quote (")
  • (","rule".*)$ - [3rd capture group] everything from ",rule" to end of line
  • \2/p - replace the line with the contents of the 2nd capture group and print
user_73661000
Related