Regex : Keep text between 2 keywords but only if another keyword exists inside them

user_13562476 0
Viewed 315

I am using emeditor and I am trying to isolate about 2 millions articles containing keyword3 from a french wikipedia dump .xml file (20GB, 338 millions rows, 4.8 millions articles in total). I would like to keep the text contained between 2 keywords (keyword1 and keyword2) but only if another keyword (keyword3) exists inside them.

List of keywords :

keyword1 = <page>
keyword2 = </page>
keyword3 = {{Infobox

Example A:

keyword1 = <page>
text to consider without keyword3
keyword2 = </page>

Result => do not extract (or keep or split) this part.

Example B:

keyword1 = <page>
text to consider with keyword3
keyword2 = </page>

Result => extract (or keep or split) this part.

The author of Emeditor helped me with the following :

Find (choose regular expression):
<page>(.*?{{Infobox.*?)</page>
Replace with
\1
And in Advanced... : search in 2500 lines

It seems to work overall fine but from time to time some errors are appearing : I am joining some tiny samples here : https://www.cjoint.com/c/JErsTJnVQpD I also added a small desired results xml file. As you can see in the joined image, the highlighted part in blue color (2 articles) should not have been included in the result part as they don't have the keyword {{Infobox . Note: It also would be nice if the tag is keep in the results. Thanks in advance ;)

emeditor notepad++ regex
3 Answers

Left in the metaphor keywords, subtitute for needed
Since have gigabytes this is fastest way to do
Try:

(?s)keyword1.*?(?:(?:keyword1|keyword2)(*SKIP)(*FAIL)|keyword3).*?(?:keyword1(*SKIP)(*FAIL)|keyword2)

demo

Or with keyword substitons:

Find (?s)<page>(.*?(?:(?:<page>|</page>)(*SKIP)(*FAIL)|{{Infobox).*?)(?:<page>(*SKIP)(*FAIL)|</page>)
Replase $1

demo

Not explain what quantifier is as some do - this is not about it
Expect to know basics

Admin1

If you use EmEditor, in the Replace dialog box:

  • Find: <page>((?:(?!<page>).)*?{{Infobox.*?)</page>
  • Replace with: <page>\1</page>

Make sure New Document is selected in the menu displayed when you click ▼ by the Extract button.

EmEditor Replace dialog box

In the Advanced dialog:

  • Set the Regular Expressions “.” Can Match Newline Characters check box.
  • Enter 3000 (or the maximum number of lines you need to extract from one occurrence of regex) at the Additional Lines to Search for Regular Expressions text box

enter image description here

Finally, click the Extract button in the Replace dialog box.

user_134849370

You need to exclude the keyword1 from matching between keyword1 and keyword3. Use

Find What: (?s)<page>((?:(?!<page>).)*?{{Infobox.*?)</page>
Replace with: \1

Here,

  • (?s) - a DOTALL modifier (same as if . matches newlines were ON)
  • <page> - keyword1 text
  • ((?:(?!<page>).)*?{{Infobox.*?) - Group 1: any char, 0 or more occurrences but as few as possible, not starting a <page> char sequence
    • (?:(?!<page>).)*?
    • {{Infobox - keyword2
    • .*? - any 0 or more chars as few as possible
  • </page> - keyword2 text
user_38329700
Related