Using variant in Bison and Flex

user_10356077 0
Viewed 978

I've been rewriting a parser from c to c++, and as such I am trying to use variant with my code. However, I am not sure how to integrate it with flex, and I keep getting esoteric error messages.

My bison file looks like

%require "3"
%language "c++"

%{
    // declarations
%}

%define api.value.type {std::variant<double, std::string>}

%token COMMENT
%token <double> DOUBLE
%token <std::string> STRING

// grammar

and my lexer looks like

%{
#include "y.tab.h"
%}
%option noyywrap

ID [a-zA-Z][a-zA-Z0-9_]*


%%
[ \t\n ]+ ;

\-?[0-9]+ |
\-?[0-9]+\. |
\-?[0-9]+\.[0-9]+ |
\-?\.[0-9]+ { yylval.emplace<double>(std::atof(yytext)); return DOUBLE;}
// other tokens
zA-Z][\.a-zA-Z0-9_]* { yylval.emplace<std::string>(yytext); return STRING;}
%%

I am not sure of my usage of yylval, I am trying to access the variant as I would have with the %union.

I get the following error:

y.tab.h:125:18: error: ‘variant’ in namespace ‘std’ does not name a template type
     typedef std::variant<double, std::string> semantic_type;
                  ^~~~~~~
y.tab.h:197:27: error: ‘semantic_type’ does not name a type
                     const semantic_type& v);
                           ^~~~~~~~~~~~~
y.tab.h:212:7: error: ‘semantic_type’ does not name a type
       semantic_type value;
       ^~~~~~~~~~~~~
my_mdl.l: In function ‘int yylex()’:
my_mdl.l:16:3: error: ‘yylval’ was not declared in this scope
 \-?\.[0-9]+ { yylval.emplace<double>(std::atof(yytext)); return DOUBLE;}
   ^~~~~~
my_mdl.l:16:3: note: suggested alternative: ‘yylex’
 \-?\.[0-9]+ { yylval.emplace<double>(std::atof(yytext)); return DOUBLE;}
   ^~~~~~
   yylex
my_mdl.l:16:18: error: expected primary-expression before ‘double’
 \-?\.[0-9]+ { yylval.emplace<double>(std::atof(yytext)); return DOUBLE;}
                  ^~~~~~
my_mdl.l:16:53: error: ‘DOUBLE’ was not declared in this scope
 \-?\.[0-9]+ { yylval.emplace<double>(std::atof(yytext)); return DOUBLE;}
                                                     ^~~~~~
my_mdl.l:18:10: error: ‘COMMENT’ was not declared in this scope
 "//".* { return COMMENT;}
          ^~~~~~~
my_mdl.l:37:29: error: expected primary-expression before ‘>’ token
 [a-zA-Z][\.a-zA-Z0-9_]* { yylval.emplace<std::string>(yytext); return STRING;}
                             ^
my_mdl.l:37:47: error: ‘STRING’ was not declared in this scope
 [a-zA-Z][\.a-zA-Z0-9_]* { yylval.emplace<std::string>(yytext); return STRING;}
                                               ^~~~~~

I also get a few hundred lines of errors from my .y file such as

my_mdl.y:88:79: error: no matching function for call to ‘MOVE::MOVE(<brace-enclosed initializer list>)’
     p.add_command(Command{in_place_index<5>, MOVE( {{$2, $3, $4}}, $5)});
                                                                               ^
In file included from parsing/symt.h:7:0,
                 from my_mdl.y:10:
parsing/cmd.h:44:5: note: candidate: MOVE::MOVE(const Scalable<double, 3>&, const string&)
     MOVE(const Scalable<double, 3> &params, const std::string &scaleFactorName);
     ^~~~

MOVE is a struct defined as

struct MOVE {
    MOVE(const Scalable<double, 3> &params, const std::string &scaleFactorName);

    Scalable<double, 3> params; // todo equationify
    std::string scale_factor_name;
};

and it is one of the types in the variant (std::variant<MOVE, etc...> Command). The strange thing is that this works normally in my code if I write p.add_command(Command{in_place_index<5>, MOVE{{{x, y, z}}, "asdfads"}});

c++17 variant lex bison c++
2 Answers

You have not included enough of your program to allow for a precise answer. Please see the SO help page on preparing a [mcse]. But it seems likely that you get the error

y.tab.h:125:18: error: ‘variant’ in namespace ‘std’ does not name a template type

because you haven't arranged for #include <variant> to be in your flex file.

The typedef itself is from the generated code in the header file produced by bison, but bison cannot guess what #include directives it might need, so it leaves it up to you to insert them. You must ensure that all types needed by your semantic type have been defined before you #include the bison-generated header. You could insert appropriate #include directives in the prologue block in your flex file, or you could use a %code requires block in your bison file. (Since you are using bison 3, the latter is probably the best solution.)

I have no idea what SAVE means in the errors from your bison file. I assume it is a macro you have (or have not defined), so the error would be a result of the macro expansion.

user_15662210

Bison actually provides its own custom type called 'variant', which is not the C++ std::variant, and supports it nicely.

https://www.gnu.org/software/bison/manual/bison.html#C_002b_002b-Variants

Bison provides a variant based implementation of semantic values for C++. This alleviates all the limitations reported in the previous section, and in particular, object types can be used without pointers.

To enable variant-based semantic values, set the %define variable api.value.type to variant (see %define Summary). Then %union is ignored; instead of using the name of the fields of the %union to “type” the symbols, use genuine types.

For instance, instead of:

%union
{
  int ival;
  std::string* sval;
}
%token <ival> NUMBER;
%token <sval> STRING;

write:

%token <int> NUMBER;
%token <std::string> STRING;

STRING is no longer a pointer, which should fairly simplify the user actions in the grammar and in the scanner (in particular the memory management).

To enable it, you simply specify

%define api.value.type variant

However this will also affect your lexer. I put together a small demo C++ flex/bison skeleton using the latest flex/bison binaries under Ubuntu:

https://github.com/kfsone/flub

highlights being:

/* lexer */
{string}            yylval->emplace<std::string>(yytext); return Token::STRING;

/* parser */
%language "C++"
%skeleton "lalr1.cc"
%require "3.8.2"

%define         api.value.type      variant
/*
%define         api.value.automove     // use with care
*/

%define         parse.assert
%define         parse.trace
%define         parse.error         detailed
%define         parse.lac           full

/* ... */

/*
 NB: You now use the TYPE rather than union-member name
*/
%token <std::string> STRING

/*
// ditto for nterms, but I now prefer putting them by the production itself.
%nterm <string::string> string_literal
*/
%%

/* ... */

/* Declare the type of this non-terminal, to be done here in the
   productions section, you need a trailing semicolon */
%nterm <std::string> using_file;
using_file
  : "using" STRING
    { $$ = $2; }
  ;

You can then use a much more modern AST-building approach, one thing I found helpful was to have a wrapper type of my own that captured a value AND its location.

template<typename ValueType>
struct ParsedType
{
  location  mLoc;  // or YourParserType::location_type
  ValueType mValue;
};

using ParsedString = ParsedType<std::string>;

/* ... */

%nterm <ParsedString> using_file;
using_file
    :    "using" STRING
         { $$ = ParsedString(@2, $2); }
user_2576450
Related