Regex to extract paragraph from a text using ruby

Viewed 90

I have PDF files from which I have to extract certain paragraph. I converted the PDF to text file using pdf-reader gem and now I am trying to extract the paragraph from the text using regular expressions.

my text looks like this after conversion

48 - Pin TSOP I (12 x 20 / 0.5 mm pitch)\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nGENERAL DESCRIPTION\n Offered in 1G x 8bit, the K9K8 is a 8G-bit NAND Flash Memo ry with spare 256M-bit. Its NAND cell provides the most cost-\n\n effective solution for the solid state application marke t. A program operation can be performed in typical 200 µs on the (2K+64)Byte\n page and an erase operation can be performed in typical 1.5ms on a (128K+4K)Byte block. Data in the data register can be read out\n at 25ns(K9NBG) cycle time per Byte. The I/O pins serve as the ports for address and data input/output as well as com-\n\n mand input. The on-chip write controller aut omates all program and erase functions in cluding pulse repetition, where required, and\n internal verification and margining of data. Even the writ e-intensive systems can take advantage of the K9K8G08U0M ′s extended\n reliability of 100K program/eras e cycles by providing ECC(Error Correc) with real time mapping-out algorithm. The\n\n K9K8G08U0M is an optimum solution for large nonvolatile storage appl ications such as solid state file storage and other portabl e\n applications requiring non-volatility.\n An ultra high density solution having two 8Gb stacked with twochip selects is also available in standard TSOPI package and another\n\n ultra high density solution having two 16Gb TSOPI package stacked with four chip selects is also available in TSOPI-DSP.\n\n\n\n\n\n\n\ntsopi dhf ghghgfhggfg hhhdhdggdj....

I want to extract the text from GENERAL DESCRIPTION to the end of the paragraph where we have multiple new lines(at least 3 \n). I have implemented following method but it is only able to extract the first line from the paragraph

def extract_desc(page_text)
 desc_headings = ['GENERAL DESCRIPTION', 'General Description',
                 'DESCRIPTION', 'Description']
 desc_regex_str = "(#{desc_headings.join('|')})[\s\n]{0,4}?((.+)\n\n)"
 desc_regex = RE2::Regexp.new(desc_regex_str)
 matched_desc_text =   desc_regex.match(page_text)
 desc = matched_desc_text[0] if matched_desc_text && matched_desc_text.size > 1
 desc
end

How can I extract the paragraph? above text is just a snippet and I have other text above and below the description. How to extract the text between ['GENERAL DESCRIPTION', 'General Description','DESCRIPTION', 'Description'] and at \n\n\n.

2 Answers

You can use

/\b(?:General Description|Description).*?(?=\n{3}|\z)/im

See the regex demo. Or, a bit faster but more verbose:

/\b(?:General Description|Description).*(?:\n(?!\n{2}).*)*/i

Details:

  • (?im) - case insensitive (i) and multiline (m, dot matches newline now) modes on
  • \b - a word boundary
  • (?:General Description|Description) - a non-capturing group matching either of the two strings, General Description or Description
  • .*? - any zero or more chars, as few as possible
  • (?=\n{3}|\z) - a positive lookahead that matches a location immediately followed with 3 newline chars or end of string.

The second pattern ends in .*(?:\n(?!\n{2}).*)* that matches

  • .* - any zero or more chars other than line breaks chars (because m modifier is absent), as many as possible
  • (?:\n(?!\n{2}).*)* - zero or more sequences of a newline not followed with two newline chars, and then the rest of the line.

See a Ruby demo:

def extract_desc(page_text)
 desc_headings = ['General Description', 'Description']
 desc_regex = /(?im)\b(?:#{desc_headings.join('|')}).*?(?=\n{3}|\z)/
 page_text[desc_regex]
end

text = "48 - Pin TSOP I (12 x 20 / 0.5 mm pitch)\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nGENERAL DESCRIPTION\n Offered in 1G x 8bit, the K9K8 is a 8G-bit NAND Flash Memo ry with spare 256M-bit. Its NAND cell provides the most cost-\n\n effective solution for the solid state application marke t. A program operation can be performed in typical 200 µs on the (2K+64)Byte\n page and an erase operation can be performed in typical 1.5ms on a (128K+4K)Byte block. Data in the data register can be read out\n at 25ns(K9NBG) cycle time per Byte. The I/O pins serve as the ports for address and data input/output as well as com-\n\n mand input. The on-chip write controller aut omates all program and erase functions in cluding pulse repetition, where required, and\n internal verification and margining of data. Even the writ e-intensive systems can take advantage of the K9K8G08U0M ′s extended\n reliability of 100K program/eras e cycles by providing ECC(Error Correc) with real time mapping-out algorithm. The\n\n K9K8G08U0M is an optimum solution for large nonvolatile storage appl ications such as solid state file storage and other portabl e\n applications requiring non-volatility.\n An ultra high density solution having two 8Gb stacked with twochip selects is also available in standard TSOPI package and another\n\n ultra high density solution having two 16Gb TSOPI package stacked with four chip selects is also available in TSOPI-DSP.\n\n\n\n\n\n\n\ntsopi dhf ghghgfhggfg hhhdhdggdj...."
p extract_desc(text)

Maybe split is enough here:


> text = "48 - Pin TSOP I (12 x 20 / 0.5 mm pitch)\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nGENERAL DESCRIPTION\n Offered in 1G x 8bit, the K9K8 is a 8G-bit NAND Flash Memo ry with spare 256M-bit. Its NAND cell provides the most cost-\n\n effective solution for the solid state application marke t. A program operation can be performed in typical 200 µs on the (2K+64)Byte\n page and an erase operation can be performed in typical 1.5ms on a (128K+4K)Byte block. Data in the data register can be read out\n at 25ns(K9NBG) cycle time per Byte. The I/O pins serve as the ports for address and data input/output as well as com-\n\n mand input. The on-chip write controller aut omates all program and erase functions in cluding pulse repetition, where required, and\n internal verification and margining of data. Even the writ e-intensive systems can take advantage of the K9K8G08U0M ′s extended\n reliability of 100K program/eras e cycles by providing ECC(Error Correc) with real time mapping-out algorithm. The\n\n K9K8G08U0M is an optimum solution for large nonvolatile storage appl ications such as solid state file storage and other portabl e\n applications requiring non-volatility.\n An ultra high density solution having two 8Gb stacked with twochip selects is also available in standard TSOPI package and another\n\n ultra high density solution having two 16Gb TSOPI package stacked with four chip selects is also available in TSOPI-DSP.\n\n\n\n\n\n\n\ntsopi dhf ghghgfhggfg hhhdhdggdj...."
> text.split(/\n{3,}/)[1]
=> "GENERAL DESCRIPTION\n Offered in 1G x 8bit, the K9K8 is a 8G-bit NAND Flash Memo ry with spare 256M-bit. Its NAND cell provides the most cost-\n\n effective solution for the solid state application marke t. A program operation can be performed in typical 200 µs on the (2K+64)Byte\n page and an erase operation can be performed in typical 1.5ms on a (128K+4K)Byte block. Data in the data register can be read out\n at 25ns(K9NBG) cycle time per Byte. The I/O pins serve as the ports for address and data input/output as well as com-\n\n mand input. The on-chip write controller aut omates all program and erase functions in cluding pulse repetition, where required, and\n internal verification and margining of data. Even the writ e-intensive systems can take advantage of the K9K8G08U0M ′s extended\n reliability of 100K program/eras e cycles by providing ECC(Error Correc) with real time mapping-out algorithm. The\n\n K9K8G08U0M is an optimum solution for large nonvolatile storage appl ications such as solid state file storage and other portabl e\n applications requiring non-volatility.\n An ultra high density solution having two 8Gb stacked with twochip selects is also available in standard TSOPI package and another\n\n ultra high density solution having two 16Gb TSOPI package stacked with four chip selects is also available in TSOPI-DSP."
Related