I need a regular expression in order to tokenize multiple words inside quotes in c++

user_4707773 0
Viewed 116

I have an input file that I want to tokenize. the text is like this:

MATERIAL  "A992Fy50"    TYPE "Steel"    GRADE "Grade 50"    WEIGHTPERVOLUME 0.007849048

The tricky part is to get the "Grade 50" as a single word. could any one suggest a regular expression in c++. I know how to do it without the white space inside the quote with:

   const std::string text{"MATERIAL  \"A992Fy50\"    TYPE \"Steel\"    GRADE \"Grade 50\"    WEIGHTPERVOLUME 0.007849048"};
    const std::regex ws_re{"\(\")?(\\s+)\(\")?"}; // whitespace
    std::copy( std::sregex_token_iterator(text.begin(), text.end(), ws_re, -1),
               std::sregex_token_iterator(),
               std::ostream_iterator<std::string>(std::cout, "\n"));

but this also splits "Grade 50" to Grade and 50.

c++ regex
4 Answers

Regex is also a text parser that is ready for use straight away, with an efficient expression that is as simple as

"([^"]*)"|(\S+)

See regex proof.

EXPLANATION

--------------------------------------------------------------------------------
  "                        '"'
--------------------------------------------------------------------------------
  (                        group and capture to \1:
--------------------------------------------------------------------------------
    [^"]*                    any character except: '"' (0 or more
                             times (matching the most amount
                             possible))
--------------------------------------------------------------------------------
  )                        end of \1
--------------------------------------------------------------------------------
  "                        '"'
--------------------------------------------------------------------------------
 |                        OR
--------------------------------------------------------------------------------
  (                        group and capture to \2:
--------------------------------------------------------------------------------
    \S+                      non-whitespace (all but \n, \r, \t, \f,
                             and " ") (1 or more times (matching the
                             most amount possible))
--------------------------------------------------------------------------------
  )                        end of \2

C++ code snippet:

#include <iostream>
#include <regex>
#include <vector>
using namespace std;

int main() {
    auto blah = std::string{"MATERIAL  \"A992Fy50\"    TYPE \"Steel\"    GRADE \"Grade 50\"    WEIGHTPERVOLUME 0.007849048"};
    std::regex reg{R"x("([^"]*)"|(\S+))x"};
    std::vector<std::string> tokens;
    std::remove_copy_if(std::sregex_token_iterator(blah.begin(), blah.end(), reg, {1, 2}), std::sregex_token_iterator(), 
            std::back_inserter(tokens),
            [](std::string const &s) { return s.empty(); });
    for (auto s: tokens) {
        std::cout << s << "\n";
    }
    return 0;
}

Results

MATERIAL
A992Fy50
TYPE
Steel
GRADE
Grade 50
WEIGHTPERVOLUME
0.007849048
user_113298900

I have a problem.

I know, I'll solve it with regular expressions!

I now have two problems; my original problem, and the regular expression.

Alternative:

  1. Break string into quote-delimited phrases. Use std::string_view::find, return a vector of struct phrase{bool quoted; std::string_view str;};

  2. Break non-quoted phrases up by whitespace.

And you are done.

Adding new stuff like escaped quotes or whitespace is easy, and your code is a bunch of C++ rather than write-only regex with unknowable runtime complexity.

Regex is great for doing a quick one-off search and replace under human supervision, but I would avoid using if for anything that matters.

user_17746670

This is best handled by std::quoted. This occurs so often that the standardization committee created a function specifically to handle quoting and unquoting stream IO. See: https://en.cppreference.com/w/cpp/io/manip/quoted

#include <iostream>
#include <iomanip>
#include <sstream>

int main()
{
    std::string s;
    std::stringstream in{ "MATERIAL  \"A992Fy50\"    TYPE \"Steel\"    GRADE \"Grade 50\"    WEIGHTPERVOLUME 0.007849048" };
    for (; in >> std::quoted(s), in;)
        std::cout << s << '\n';
};

Output:

MATERIAL
A992Fy50
TYPE
Steel
GRADE
Grade 50
WEIGHTPERVOLUME
0.007849048
user_52821540

Instead of focusing on finding a token iterator, I focused on a regular expression to match each whole input:

std::string s = R"(MATERIAL  "A992Fy50"    TYPE "Steel"    GRADE "Grade 50"    WEIGHTPERVOLUME 0.007849048)";

std::regex e{R"((\w+)\s+("[^"]*"|\d\.\d+))"};

std::sregex_iterator m_end;
std::sregex_iterator m(std::begin(s), std::end(s), e);

for(; m != m_end; ++m)
{
    std::cout << "Full match: "<< m->str(0) << '\n';

    for(auto i = 1U; i < m->size(); ++i)
        std::cout << "    sub " << i << ": " << m->str(i) << '\n';
}

Output:

Full match: MATERIAL  "A992Fy50"
    sub 1: MATERIAL
    sub 2: "A992Fy50"
Full match: TYPE "Steel"
    sub 1: TYPE
    sub 2: "Steel"
Full match: GRADE "Grade 50"
    sub 1: GRADE
    sub 2: "Grade 50"
Full match: WEIGHTPERVOLUME 0.007849048
    sub 1: WEIGHTPERVOLUME
    sub 2: 0.007849048
user_38077290
Related