Regexp for URL validation

user_13214408 0
Viewed 262

I am writing my own http server. Checking and validating headers and values ​​from requests, oddly enough, is one of its tasks. I found a page with a table comparing different regular expressions and decided to use one of them:

_^(?:(?:https?|ftp)://)(?:\S+(?::\S*)?@)?(?:(?!10(?:\.\d{1,3}){3})(?!127(?:\.\d{1,3}){3})(?!169\.254(?:\.\d{1,3}){2})(?!192\.168(?:\.\d{1,3}){2})(?!172\.(?:1[6-9]|2\d|3[0-1])(?:\.\d{1,3}){2})(?:[1-9]\d?|1\d\d|2[01]\d|22[0-3])(?:\.(?:1?\d{1,2}|2[0-4]\d|25[0-5])){2}(?:\.(?:[1-9]\d?|1\d\d|2[0-4]\d|25[0-4]))|(?:(?:[a-z\x{00a1}-\x{ffff}0-9]+-?)*[a-z\x{00a1}-\x{ffff}0-9]+)(?:\.(?:[a-z\x{00a1}-\x{ffff}0-9]+-?)*[a-z\x{00a1}-\x{ffff}0-9]+)*(?:\.(?:[a-z\x{00a1}-\x{ffff}]{2,})))(?::\d{2,5})?(?:/[^\s]*)?$_iuS

then i went to https://regex101.com and tried to test it.
After some changes:

^(?:(?:https?|ftp):\/\/)(?:\S+(?::\S*)?@)?(?:(?!10(?:\.\d{1,3}){3})(?!127(?:\.\d{1,3}){3})(?!169\.254(?:\.\d{1,3}){2})(?!192\.168(?:\.\d{1,3}){2})(?!172\.(?:1[6-9]|2\d|3[0-1])(?:\.\d{1,3}){2})(?:[1-9]\d?|1\d\d|2[01]\d|22[0-3])(?:\.(?:1?\d{1,2}|2[0-4]\d|25[0-5])){2}(?:\.(?:[1-9]\d?|1\d\d|2[0-4]\d|25[0-4]))|(?:(?:[a-z\x{00a1}-\x{ffff}0-9]+-?)*[a-z\x{00a1}-\x{ffff}0-9]+)(?:\.(?:[a-z\x{00a1}-\x{ffff}0-9]+-?)*[a-z\x{00a1}-\x{ffff}0-9]+)*(?:\.(?:[a-z\x{00a1}-\x{ffff}]{2,})))(?::\d{2,5})?(?:\/[^\s]*)?$

if passes my "test":

http://userid:password@example.com:8080
http://userid:password@example.com/
http://example.com/blah_blah
http://example.com/blah_blah_(wikipedia)
http://userid:password@example.com/
http://example.com/blah_(wikipedia)_blah#cite-1
http://example.bar/?q=Test%20URL-encoded%20stuff
http://example.bar?q=Spaces should be encoded
http://a.b--c.de/

But i need it inside my c++ server, so i again modified it :

std::regex re(
               "^(?:(?:https?|ftp):\\/\\/)(?:\\S+(?::\\S*)?@)?(?:(?!10(?:\\.\\d{1,3})"
               "{3})(?!127(?:\\.\\d{1,3}){3})(?!169\\.254(?:\\.\\d{1,3}){2})(?!"
               "192\\.168(?:\\.\\d{1,3}){2})(?!172\\.(?:1[6-9]|2\\d|3[0-1])(?:\\.\\d{"
               "1,3}){2})(?:[1-9]\\d?|1\\d\\d|2[01]\\d|22[0-3])(?:\\.(?:1?\\d{1,2}|2["
               "0-4]\\d|25[0-5])){2}(?:\\.(?:[1-9]\\d?|1\\d\\d|2[0-4]\\d|25[0-4]))|(?"
               ":(?:[a-z\\x{00a1}-\\x{ffff}0-9]+-?)*[a-z\\x{00a1}-\\x{ffff}0-9]+)(?"
               ":\\.(?:[a-z\\x{00a1}-\\x{ffff}0-9]+-?)*[a-z\\x{00a1}-\\x{ffff}0-9]+)"
               "*(?:\\.(?:[a-z\\x{00a1}-\\x{ffff}]{2,})))(?::\\d{2,5})?(?:\\/[^\\s]*)"
               "?$");

with edition above my compiler warns: invalid hexadecimal number and if i run i get

terminate called after throwing an instance of 'std::regex_error'
  what():  Unexpected end of regex when ascii character.

the error as said in that warning is in \\x{00a1}-\\x{ffff} but I don't know regex well enough to fix that. Please help, thank you.

http c++ regex
1 Answers

You need to replace all \x{XXXX} with \uXXXX since ECMAScript regex flavor does not support \x{XXXX} notation. Next time you use regex online testers, make sure you select the appropriate regex flavor (C++ uses ECMAScript by default, not PCRE).

Next, you can also replace \/ with / (you only need \/ at the online regex tester when / chars are used as regex delimiters, or in languages where regexps are defined with the help of regex literals that are delimited with /), and you may shorten [^\s] to \S.

Next, (?::\S*)? close to the beginning of the pattern is redundant, it can be safely removed, and the (?:[a-z\x{00a1}-\x{ffff}0-9]+-?)* parts in the regex will cause catastrophic backtracking. To avoid it, the ? quantifier after - must be removed.

So, the pattern you want to use is

^(?:(?:https?|ftp):\/\/)(?:\S+@)?(?:(?!10(?:\.\d{1,3}){3})(?!127(?:\.\d{1,3}){3})(?!169\.254(?:\.\d{1,3}){2})(?!192\.168(?:\.\d{1,3}){2})(?!172\.(?:1[6-9]|2\d|3[0-1])(?:\.\d{1,3}){2})(?:[1-9]\d?|1\d\d|2[01]\d|22[0-3])(?:\.(?:1?\d{1,2}|2[0-4]\d|25[0-5])){2}(?:\.(?:[1-9]\d?|1\d\d|2[0-4]\d|25[0-4]))|(?:[a-z\u00a1-\uffff0-9]+-)*[a-z\u00a1-\uffff0-9]+(?:\.(?:[a-z\u00a1-\uffff0-9]+-)*[a-z\u00a1-\uffff0-9]+)*(?:\.(?:[a-z\u00a1-\uffff]{2,})))(?::\d{2,5})?(?:\/\S*)?$

See the regex demo.

Now, this is how you can use it in the code:

#include <string>
#include <iostream>
#include <regex>
using namespace std;

int main() {
    std::regex r(R"(^(?:(?:https?|ftp)://)(?:\S+@)?(?:(?!10(?:\.\d{1,3}){3})(?!127(?:\.\d{1,3}){3})(?!169\.254(?:\.\d{1,3}){2})(?!192\.168(?:\.\d{1,3}){2})(?!172\.(?:1[6-9]|2\d|3[0-1])(?:\.\d{1,3}){2})(?:[1-9]\d?|1\d\d|2[01]\d|22[0-3])(?:\.(?:1?\d{1,2}|2[0-4]\d|25[0-5])){2}(?:\.(?:[1-9]\d?|1\d\d|2[0-4]\d|25[0-4]))|(?:[a-z\u00a1-\uffff0-9]+-)*[a-z\u00a1-\uffff0-9]+(?:\.(?:[a-z\u00a1-\uffff0-9]+-)*[a-z\u00a1-\uffff0-9]+)*(?:\.(?:[a-z\u00a1-\uffff]{2,})))(?::\d{2,5})?(?:/\S*)?$)");
    std::vector<std::string> lst;
    lst.push_back("http://userid:password@example.com:8080");
    lst.push_back("http://userid:password@example.com/");
    lst.push_back("http://example.com/blah_blah");
    lst.push_back("http://example.com/blah_blah_(wikipedia)");
    lst.push_back("http://userid:password@example.com/");
    lst.push_back("http://example.com/blah_(wikipedia)_blah#cite-1");
    lst.push_back("http://example.bar/?q=Test%20URL-encoded%20stuff");
    lst.push_back("http://example.bar?q=Spaces should be encoded");
    lst.push_back("http://a.b--c.de/");
    
    for (auto s : lst) {
        if (std::regex_match(s, r)) {
            std::cout << "MATCH: " << s << '\n';
        } else {
            std::cout << "NO MATCH: " << s << '\n';
        }
    }

    return 0;
}

Output of the program:

MATCH: http://userid:password@example.com:8080
MATCH: http://userid:password@example.com/
MATCH: http://example.com/blah_blah
MATCH: http://example.com/blah_blah_(wikipedia)
MATCH: http://userid:password@example.com/
MATCH: http://example.com/blah_(wikipedia)_blah#cite-1
MATCH: http://example.bar/?q=Test%20URL-encoded%20stuff
NO MATCH: http://example.bar?q=Spaces should be encoded
NO MATCH: http://a.b--c.de/
user_38329700
Related