Grep and Sed Equivalent for XML Command Line Processing

Viewed 73769

When doing shell scripting, typically data will be in files of single line records like csv. It's really simple to handle this data with grep and sed. But I have to deal with XML often, so I'd really like a way to script access to that XML data via the command line. What are the best tools?

14 Answers

Some promising tools:

  • nokogiri: parsing HTML/XML DOMs in ruby using XPath & CSS selectors

  • hpricot: deprecated

  • fxgrep: Uses its own XPath-like syntax to query documents. Written in SML, so installation may be difficult.

  • LT XML: XML toolkit derived from SGML tools, including sggrep, sgsort, xmlnorm and others. Uses its own query syntax. The documentation is very formal. Written in C. LT XML 2 claims support of XPath, XInclude and other W3C standards.

  • xmlgrep2: simple and powerful searching with XPath. Written in Perl using XML::LibXML and libxml2.

  • XQSharp: Supports XQuery, the extension to XPath. Written for the .NET Framework.

  • xml-coreutils: Laird Breyer's toolkit equivalent to GNU coreutils. Discussed in an interesting essay on what the ideal toolkit should include.

  • xmldiff: Simple tool for comparing two xml files.

  • xmltk: doesn't seem to have package in debian, ubuntu, fedora, or macports, hasn't had a release since 2007, and uses non-portable build automation.

xml-coreutils seems the best documented and most UNIX-oriented.

To Joseph Holsten's excellent list, I add the xpath command-line script which comes with Perl library XML::XPath. A great way to extract information from XML files:

 xpath -q -e '/entry[@xml:lang="fr"]' *xml

Depends on exactly what you want to do.

XSLT may be the way to go, but there is a learning curve. Try xsltproc and note that you can hand in parameters.

XQuery might be a good solution. It is (relatively) easy to learn and is a W3C standard.

I would recommend XQSharp for a command line processor.

Grep Equivalent

You can define a bash function, say "xp" ("xpath") that wraps some python3 code. To use it you need to install python3 and python-lxml. Benefits:

  1. regex matching which you lack in e.g. xmllint.
  2. Use as a filter (in a pipe) on the commandline

It's easy and powerful to use like this:

xmldoc=$(cat <<EOF
<?xml version="1.0" encoding="utf-8"?>
<job xmlns="http://www.sample.com/">programming</job>
EOF
)
selection='//*[namespace-uri()="http://www.sample.com/" and local-name()="job" and re:test(.,"^pro.*ing$")]/text()'
echo "$xmldoc" | xp "$selection"
# prints programming

xp() looks something like this:

xp()
{ 
local selection="$1";
local xmldoc;
if ! [[ -t 0 ]]; then
    read -rd '' xmldoc;
else
    xmldoc="$2";
fi;
python3 <(printf '%b' "from lxml.html import tostring\nfrom lxml import etree\nfrom sys import stdin\nregexpNS = \"http://exslt.org/regular-expressions\"\ntree = etree.parse(stdin)\nfor e in tree.xpath('""$selection""', namespaces={'re':regexpNS}):\n  if isinstance(e, str):\n    print(e)\n  else:\n    print(tostring(e).decode('UTF-8'))") <<< "$xmldoc"
}

Sed Equivalent

Consider using xq which gives you the full power of the jq "programming language". If you have python-pip installed, you can install xq with pip install yq, then in below example we are replacing "Keep Accounts" with "Keep Accounts 2":

xmldoc=$(cat <<'EOF'
<resources>
    <string name="app_name">Keep Accounts</string>
    <string name="login">"login"</string>
    <string name="login_password">"password:"</string>
    <string name="login_account_hint">input to login</string>
    <string name="login_password_hint">input your password</string>
    <string name="login_fail">login failed</string>
</resources>
EOF
)
echo "$xmldoc" | xq '.resources.string = ([.resources.string[]|select(."#text" == "Keep Accounts") ."#text" = "Keep Accounts 2"])' -x

JEdit has a plugin called "XQuery" which provides querying functionality for XML documents.

Not quite the command line, but it works!

Related