Extract multiline XML from log based on value in tag

user_10456036 0
Viewed 112

I have a log file on unix host (quite big one >500 MB) on daily basis with printed out XMLs there that are sent to other systems.

The XML structure is likely to the following one:

2021-01-04 08:44:43,174 Not related message
2021-01-04 08:45:45,174 [Sender] INFO  com....router.MessageRouter - Publishing Message: <?xml version="1.0" encoding="UTF-8"?>
  <Q1:book>
    <title lang="en">Everyday Italian</title>
    <author>Giada De Laurentiis</author>
    <year>2005</year>
    <price>30.00</price>
    <id>6AA2BC</id>
    <metadata>
       <isbn>123-432-234</isbn>
       <catalogId>111</catalogId>
    </metadata>
    <relatedBook>
       <metabook id="3AA345">Everyday Italian. Part 2</metabook>
       <metabook id="7ANNN2">Everyday Italian. Part 3</metabook>
    </relatedBook>
  </Q1:book>
2021-01-04 08:46:23,174 Not related message
2021-01-04 08:46:51,174 Not related message
2021-01-04 08:47:21,174 Not related message
2021-01-04 08:49:45,174 Message: <?xml version="1.0" encoding="UTF-8"?>
  <Q1:book>
    <title lang="en">Everyday Italian. Part 2</title>
    <author>Giada De Laurentiis</author>
    <year>2005</year>
    <price>30.00</price>
    <id>3AA345</id>
    <metadata>
       <isbn>123-432-235</isbn>
       <catalogId>115</catalogId>
    </metadata>
    <relatedBook>
       <metabook id="6AA2BC">Everyday Italian. Part 2</metabook>
       <metabook id="7ANNN2">Everyday Italian. Part 3</metabook>
    </relatedBook>
  </Q1:book>

and there are dozens of XMLs in log like that.

I have id of the my desired xml (e.g. 6AA2BC) and I need to extract the full XML related to it using Linux commands.

if I do something like grep 6AA2BC file.log I will receive the only line <id>3AA345</id>

I have tried different approaches described by community and was able to get ALL XMLs using

grep -Poz '(?<=<Q1:book)(.*?\n)*?.*?(?=</Q1:book>)' file.log but if I add id to the request, it does not work

The expected output: full XML with that particular id=6AA2BC

<Q1:book>
    <title lang="en">Everyday Italian</title>
    <author>Giada De Laurentiis</author>
    <year>2005</year>
    <price>30.00</price>
    <id>6AA2BC</id>
    <metadata>
       <isbn>123-432-234</isbn>
       <catalogId>111</catalogId>
    </metadata>
    <relatedBook>
       <metabook id="3AA345">Everyday Italian. Part 2</metabook>
       <metabook id="7ANNN2">Everyday Italian. Part 3</metabook>
    </relatedBook>
  </Q1:book>

Can anyone, please, explain what's the best way to do this?

grep awk linux xml regex
3 Answers

Another one tested with GNU awk

awk '/<Q1/{f=1}f{i = i $0 ORS }/<\/Q1/{if(i~/<id>6AA2BC/){printf "%s", i} i=f="" }' file.log
user_88608650

Supposing you have the Perl script xpath installed on your Linux system, you could do the following:

To extract the node Q1:book having the text string 6AA2BC in the subnode id from your input above, you could use:

/usr/bin/xpath -q -e '//Q1:book[id/text()="6AA2BC"]' file.log

But... first, you have to sanitize your input, and add a root opening tag in the beginning and the corresponding closing tag at the end.

As the XML lines begin with spaces, and the other lines without, keep the XML lines using the filter:

sed '/^[^ ]/d'

Add an opening root tag that must also include the definition of the Q1 name space in order to work with the commands using XPATH selection:

echo '<?xml version="1.0" encoding="UTF-8"?>'
echo '<root xmlns:Q1="https://example.com/mynamespace">'

in the beginning, and:

echo "</root>"

at the end.

Combining the above elements gives us the next one liner in Linux, that I will split on several lines for the sake of clarity

( echo '<?xml version="1.0" encoding="UTF-8"?>'
  echo '<root xmlns:Q1="https://example.com/mynamespace">'
  cat file.log | sed '/^[^ ]/d'
  echo "</root>"
) | /usr/bin/xpath -q -e '//Q1:book[id/text()="6AA2BC"]'

If you have xmllint on your system instead of xpath, just issue:

| /usr/bin/xmllint --xpath '//*[local-name()="book" and id/text()="6AA2BC"]' -

as last part of the command.

user_17827820

A proper xml parser like xmlstarlet or xmllint is the best solution for XML but given there is none xml text in the file, awk is an alternative:

awk 'BEGIN { RS="(</Q1:book>)|(<Q1>)" } /<id>6AA2BC<\/id>/ { print "<Q1>";print $0;print "</Q1:book>" }' file

Set the record separator to "" or "</Q1:book>" and print the record along with the starting and ending Q1 tags when id is equal to 6AA2BC

user_78404400
Related