Tokens in C++ and how they are used

Question

In Bjarne Stroustrup's book "Programming Principles and Practices using C++", he is explaining how you can use tokens to stop C++ from automatically using the order of operations on a simple calculator. The code he has given in book does not work - either because of my stupidity, or because I am missing something. I understand that tokens is essentially breaking down lines of code into characters. For example, If I have 5 * 3 there are 3 tokens. Two values and one character. I do not know how to incorporate tokens, or why we should use them. Bjarne's example of a user defined token is this:

class Token {
    public:
        char kind; // what kind of token
        double value; // for numbers - a value
        Token(char ch) // make a token from a char using a constructor
            :kind(ch), value(0) {} // set kind to ch and value to val
        Token(char ch, double val) // make a Token from a char and a double
            :kind(ch), value(val) {}
};

In addition he has provided this example of how you can read input into a vector of tokens:

Token get_token() {} // read a token from cin

vector tok; // put the tokens in this vector

while(cin >> t5) {
        Token t5 = get_token();
        tok.push_back(t5);
    }
    return 0;

First of all... 1) He prototyped the "get_token()" function but he didn't even write any code for it. 2) Token's object 't5' is getting initialized after it's already getting read in a while loop. 3) I have tried many ways of getting cin to read input into t5, and it is not working. I even defined t5 before the while loop, and I'm getting an invalid operand error for '>>'. This is the first time in this book that I've been utterly stuck. I can't find any examples online of how you can use tokens in C++ to write programs, I only get vague definitions of what a token is. If anyone could assist me in understanding tokens or possibly point me to a good source where I can get a thorough explanation, it would be greatly appreciated.

user_9612098 · Answer

Here you are the answer to your question #1. This is the get_token() function you were asking for!

Here is a link for it below (from Stroustrup's website directly :)):

   Token get_token()    // read a token from cin
    {
        char ch;
        cin >> ch;    // note that >> skips whitespace (space, newline, tab, etc.)

        switch (ch) {
     //not yet   case ';':    // for "print"
     //not yet   case 'q':    // for "quit"
        case '(': case ')': case '+': case '-': case '*': case '/': 
            return Token(ch);        // let each character represent itself
        case '.':
        case '0': case '1': case '2': case '3': case '4':
        case '5': case '6': case '7': case '8': case '9':
            {    
                cin.putback(ch);         // put digit back into the input stream
                double val;
                cin >> val;              // read a floating-point number
                return Token('8',val);   // let '8' represent "a number"
            }
        default:
            error("Bad token");
        }
    }

Tokens in C++ and how they are used

2 Answers