Make regex match past line end

user_657477 0
Viewed 143

I am trying to build a regex to help me parse lots of robots.txt files. I am trying to find if specific user-agents have been disallowed.

Files look roughly like his but can vary a bit with whitepace:

User-agent: *
Disallow: /next/

User-agent: *
Disallow: /mshots/v1/

# har har
User-agent: *
Disallow: /activate/

User-agent: *
Disallow: /wp-login.php

User-agent: *
Disallow: /signup/

User-agent: *
Disallow: /related-tags.php

So far I have come up with this regex:

user-agent.*:(.*$)

With options set for multiline and ignorecase.

This works great for the user-agent part of my requirement but now I need to match any dissallow word that comes before another user-agent is declared. I have tried matching $ and \r\n but I can't figure out how to keep the match going after the $. Any pattern I add after doesn't seem to work. For instance:

user-agent.*:(.*$)(^.*disallow)

I am trying to match the newline and then dissallow but it doesn't match. Also I am not sure how to terminate the match before the user-agent and still work on last declaard rule.

Can anyone point me in right direction?

regex c#
3 Answers

You may use

(?i)user-agent.*:(.*)([\r\n]\s*.*disallow)

See the regex demo. You do not need a RegexOptions.Multiline option here, because there is no need using ^, nor $ that would need redefining their behavior of.

Details

  • (?i) - case insensitive modifier
  • user-agent - a literal substring
  • .* - any 0+ chars other than a newline, as many as possible
  • : - a colon
  • (.*) - Group 1: any 0+ chars other than a newline up to the end of the line
  • ([\r\n]\s*.*disallow) - Group 2:
    • [\r\n]\s* - a line break char (CR or LF) and then 0+ whitespaces
    • .* - any 0+ chars other than a newline, as many as possible
    • disallow - a substring
user_38329700

Your problem seems to be that your are not matching the newline between the User-agent: * and Disallow.

A regex like this should work: /^user-agent: (.*)$[\s]^disallow: (.*)$/igm

This regex will capture the value of User-agent and the value of Disallow, and will work with any type of newline. The igm flags mean case-Insensitive, Global and Multiline

You can test it using regexr.

user_39635610

You can use the SingleLine option and the following regular expression:

    String pattern = @"user-agent[^\r\n]*: ([^\r\n]+)[\r\n]+disallow[^\r\n]*: ([^\r\n]+)[\r\n]+";

    foreach (Match match in Regex.Matches(robots, pattern, RegexOptions.IgnoreCase | RegexOptions.Singleline))
    {
        Console.Out.WriteLine("---------");
        Console.Out.WriteLine("Match: `" + match.ToString() + "`");
        foreach (Group grp in match.Groups)
        {
            Console.Out.WriteLine("    Group: ``" + grp.ToString() + "``");
        }               
    }

Breaking down the pattern:

A literal user-agent followed by any number of non-CRLF characters followed gy a colon and then a space. user-agent[^\r\n]*:

At least one non-CRLF character as a group. ([^\r\n]+)

At least one CRLF character. [\r\n]+

A literal disallow followed by any number of non-CRLF characters followed gy a colon and then a space. disallow[^\r\n]*:

At least non-CRLF characters as a group. ([^\r\n]+)

At least one CRLF character. [\r\n]+

The SingleLine option means "treat the input as a single line."

user_11027260
Related