issue related to splitting string by whitespace

user_19114178 0
Viewed 41

i am currently making a "stack-based programming language" where in order to push a string onto the stack, you shall use " at the beggining on the string, and also on the end. So in the end an acceptable string is something like this: "some_string".

However, i need to tokenize the whole line of the file i am reading (splitting by whitespaces). This means i cant use such string: "hello are you there?" because it would get splitten up into "hello (new element) are (new element) you (new element) there?". And this is a problem because i want it to be a whole element, not four separated elements (since texts that are not beggining with a " are treated as keywords, and it would lead to a keyword not found error because i dont have the keyword yes).

This is my current code (where str should be the line i am reading from the file, and the delimiter is a simple whitespace, ' '):

std::vector<std::string> tokenize(std::string str, char delim) {
    std::stringstream ss(str);
    std::vector<std::string> out;

    std::string s;
    while (std::getline(ss, s, delim)) {
        s.erase(std::remove(s.begin(), s.end(), '\t'), s.end());
        out.push_back(s);
    }

    return out; }

What approach shall i take?

c++
1 Answers

std::quoted will be your friend. Please read here about it.

Then your task will be as simple as the below:

#include <iostream>
#include <iomanip>
#include <string>
#include <sstream>
#include <vector>

std::string test{ "abc  \t def \"ghi jk\"\t  lmn\top\n" };

std::vector<std::string> split(std::string& s) {

    std::istringstream iss(s);

    std::string part{};
    std::vector<std::string> result{};

    while (iss >> std::quoted(part))
        result.push_back(std::move(part));

    return result;
}

int main() {
    auto parts = split(test);
    for (const auto& part : parts)
        std::cout << part << '\n';
}
user_96660180
Related