Why do dashes mess up regex search?

Viewed 77

I have 150 files each with a header similar to this one

---
layout: page
title:  "WE07S-AWE"
date:  2018-10-21 01:31:26.000000000 -0600
---

Lots of text here...

where I would like to extract the title WE07S-AWE and everything after the second line of dashes.

However the dashes messes up the search somehow.

$ ruby -pe 's/---.*title:  "(.*?)".*---(.*)/m' test
-e:1: syntax error, unexpected '.'
s/---.*title:  "(.*?)".*---(.*)/m
-e:1: syntax error, unexpected '.'
s/---.*title:  "(.*?)".*---(.*)/m

Question

Can anyone figure out what I am doing wrong?

3 Answers

You can use

ruby -0777 -pe 'sub(/---.*title:\s*"(.*?)".*---(.*)/m, "\\1 \\2")' test
ruby -0777 -ne 'print $1 + $2 if /---.*title:\s*"(.*?)".*---(.*)/m' test

Details

  • -0777 will slurp the file into a single string
  • m modifier makes . match line break chars, too, that it does not match by default
  • Instead of a single literal space between title: and ", a \s* is used to match zero or more whitespaces
  • sub will replace the match with the contents of Group 1 + Group 2, \1 \2.
  • -n assumes 'while gets(); ... end' loop around your script
  • print $1 + $2 if /---.*title:\s*"(.*?)".*---(.*)/m will print Group 1 + Group 2 value if regex matches.

IMHO, you should avoid the .* pattern when possible. Not only could it be a high performance risk, it often doesn't really describe what you're looking for.

For example, if your title never escapes the " character, a better approach might be:

 /title:[\s]+"([^"]+)".*---(.*)/m

The rest of the details are better answered by Wiktor Stribiżew (who's answer I will upvote).

Suppose str = File.read(<filename>), where

str = <<~BITTER_END
---
layout: page
title:  "WE07S-AWE"
date:  2018-10-21 01:31:26.000000000 -0600
---
Humpty Dumpty sat
on a wall
---
layout: page
title:  "WE08RS-WEA"
date:  2018-10-22 07:31:26.000000000 -0600
---
Little Miss
Muffet sat on
her tuffet
---
layout: page
title:  "AR91G-HUH"
date:  2017-03-13 01:30:26.000000000 -0800
---
Three blind mice
See how they run
BITTER_END

Rather than using a single regular expression you may wish perform a sequence of operations, both to improve readability and facilitate testing. I will use two regular expressions:

r1 = /^---\r?\n/
r2 = /^title: +"([^"]+)/

r1 reads, "match a line comprised of three hyphens". ^ is the beginning-of-line anchor and \r?\n is the line terminator (which optionally includes the carriage return character \r in case the file was created using Windows).

r2 reads "match 'title:' at the beginning of a line followed by one or more spaces (+), a double-quote, followed by one or more characters other than a double-quote (as many as possible). [^"] is a character class that matches any character other than ".

We may then write:

str.split(r1).
    drop(1).
    each_slice(2).
    with_object({}) { |(header,body),h| h[header[r2,1]] = body }
  #=> {"WE07S-AWE"=>"Humpty Dumpty sat\non a wall\n",
  #    "WE08RS-WEA"=>"Little Miss\nMuffet sat on\nher tuffet\n",
  #    "AR91G-HUH"=>"Three blind mice\nSee how they run\n"}

The steps are as follows.

a = str.split(r1)
  #=> ["",
  #    "layout: page\ntitle:  \"WE07S-AWE\"\ndate:  2018-10-21 01:31:26.000000000 -0600\n",
  #    "Humpty Dumpty sat\non a wall\n",
  #    "layout: page\ntitle:  \"WE08RS-WEA\"\ndate:  2018-10-22 07:31:26.000000000 -0600\n",
  #    "Little Miss\nMuffet sat on\nher tuffet\n",
  #    "layout: page\ntitle:  \"AR91G-HUH\"\ndate:  2017-03-13 01:30:26.000000000 -0800\n",
  #    "Three blind mice\nSee how they run\n"] 
b = a.drop(1)
  #=> ["layout: page\ntitle:  \"WE07S-AWE\"\ndate:  2018-10-21 01:31:26.000000000 -0600\n",
  #   ...
  #    "Three blind mice\nSee how they run\n"] 
c = b.each_slice(2)
  #=> #<Enumerator: ["layout: page\ntitle:  \"WE07S-AWE\"\ndate:  2018-10-21 01:31:26.000000000 -0600\n",..., "Three blind mice\nSee how they run\n"]:each_slice(2)> 

We can see the elements that will be generated by the enumerator c and passed to with_object by converting it to an array.

c.to_a
  #=> [["layout: page\ntitle:  \"WE07S-AWE\"\ndate:  2018-10-21 01:31:26.000000000 -0600\n",
  #     "Humpty Dumpty sat\non a wall\n"],
  #    ["layout: page\ntitle:  \"WE08RS-WEA\"\ndate:  2018-10-22 07:31:26.000000000 -0600\n",
  #     "Little Miss\nMuffet sat on\nher tuffet\n"],
  #    ["layout: page\ntitle:  \"AR91G-HUH\"\ndate:  2017-03-13 01:30:26.000000000 -0800\n",
  #     "Three blind mice\nSee how they run\n"]] 

Continuing,

d = c.with_object({})
  #=> #<Enumerator: #<Enumerator: ["layout:...]:each_slice(2)>:each_with_object({"\"WE07S-AWE"=>"Humpty Dumpty sat\non a wall\n"})> 

d may be thought of as a compound enumerator though Ruby has no such concept. Continuing,

(header,body),h = d.next
  #=> [["layout: page\ntitle:  \"WE07S-AWE\"\ndate:  2018-10-21 01:31:26.000000000 -0600\n",
  #      "Humpty Dumpty sat\non a wall\n"],
  #    {}]

Ruby uses array decomposition to break d.next into three objects which respectively become values of the three block variables header, body and h. Let's examine those values.

header
  #=> "layout: page\ntitle:  \"WE07S-AWE\"\ndate:  2018-10-21 01:31:26.000000000 -0600\n" 
body
  #=> "Humpty Dumpty sat\non a wall\n" 
h #=> {} 

This is the initial value of h. It will be built in the course of the calculations. Now examine the block calculation.

s = header[r2,1]
  #=> "WE07S-AWE"
h[s] = body
  #=> "Humpty Dumpty sat\non a wall\n"

so now

h #=> {"WE07S-AWE"=>"Humpty Dumpty sat\non a wall\n"}

The remaining calculations are similar.

See String#split, Array#drop, Enumerable#each_slice and Enumerator#with_object.

Related