yytext on lex is accumulating words

Viewed 34

I made a simple program that reads one or more files and then map the words and how many times these words appear (on both files). The problem is that I don't know why it is accumulating words on the yytext.

It should be like: cat line 5 line 4 file 1

but It cannot even enter on the if condition where it compares the word(s) present on the table and the current.

It's somehow accumulating words like "cat owns" should be the word cat on one node and another one in the other to compare cat and owns with strcmp, but it's comparing "owns" with "cat owns".

My code

%option noyywrap
%option nodefault
%option yylineno
%option warn
%option case-insensitive

%{
  #include <stdlib.h>
  #include <stdio.h>
  #include <string.h>

  typedef struct _la{      /* no da lista de ocorrencias */
    int line;
    char * arq;
    struct _la * next;
  } LineArq;

  typedef struct _node {    /* entrada na tabela de simbolos */
   char * word;
   LineArq *lines;
   struct _node * next;
  } Node;


  char *curfilename;

  Node * tabela = NULL;

  Node * getNodeL( char * word, int line, char * arq );
  void insertNode(char * word, int line, char * arq, Node **l);
  void show(Node *l);
%}

%%

a |
an |
and |
are |
as |
at |
be |
but |
for |
in |
is |
it |
of |
on |
or |
that |
the |
this |
to               { /* printf("DEBUG: [%d, %s] %s\n", yylineno, curfilename, yytext ); */ }  /* ESSES PADROES SAO IGNORADOS... */

[a-z]+(\'(s|t))?     { insertNode(yytext, yylineno, curfilename, &tabela);
                    /* printf("DEBUG: [%d, %s] %s\n", yylineno, curfilename, yytext ); */
                 }

.|\n             { /* printf("DEBUG: [%d, %s] %s\n", yylineno, curfilename, yytext ); */ }  /* IGNORA NOVA LINHA E QUALQUER OUTRO CARACTER... */

%%

int main(int argc, char *argv[]){
  int i = 0;

  for(i = 1; i < argc; i++) {
    FILE *f = fopen(argv[i], "r");
    if(!f) {
      perror(argv[i]);
      return (1);
    }
    curfilename = argv[i]; 
    yyrestart(f);
    yylineno = 1;
    yylex();
    fclose(f);
  }

  show(tabela);
}

Node * getNodeL(char * word, int line, char * arq){
  
  Node * tmp = (Node *) malloc(sizeof(Node));

  if ( tmp == NULL ) {
    printf("Erro ao alocar memoria para o no da lista\n");
    exit( 1023 ); /* documentar no Dicionario de dados */
  }

  LineArq * la = (LineArq *) malloc(sizeof(LineArq));

  if (la == NULL){
    printf("Erro ao alocar memoria para o no da lista\n");
    exit( 1024 ); /* documentar no Dicionario de dados */
  }

  la->line = line;
  la->arq = arq;
  la->next = NULL;
  
  tmp->word = word;
  tmp->lines = la;
  tmp->next = NULL;

  return tmp;
}

void insertNode(char * word, int line, char * arq, Node **l){


  printf("[DEBUG] >>>>>>> %s     %d \t %s\n", word, line, arq);

  Node * p = *l;
  int count = 0;
  
  /* primeira entrada na tabela */
  if ( p == NULL ) {
    printf("entrou\n");
    *l = getNodeL(word, line, arq);
    return;
  }
  printf("%s\n", p->word);

  while( p != NULL ) {
     printf("[DEBUG]       [%s] == [%s]\n", word, p->word);
     if ( strcmp(word, p->word) == 0 ) {
        printf("entrou palavra igual\n");
        break;
     }
     p = p-> next;
  }

  if ( p != NULL ) { /* palavra encontrada na tabela */
      LineArq * la = (LineArq *) malloc(sizeof(LineArq));
      la->line = line;
      la->arq = arq;
      p->lines->next = la;
      la->next = NULL;
  } else {
    Node * new = getNodeL(word, line, arq);
    new->next = *l;
    *l = new;
  }
}

void show(Node *l){
  Node * tmp = l;
  LineArq * linesA;
  while( tmp != NULL ) {
    printf("Palavra: %10s\n", tmp->word);
    linesA = tmp->lines;
    while (linesA != NULL){
      printf("\tLinha: %d \t Arquivo: %s\n", linesA->line, linesA->arq);
      linesA = linesA->next;
    }
    /*
    if ( tmp->next != NULL ) {
        printf(", ");
    }
    */
    tmp = tmp->next;
  }
}

And what is coming out:

because the cat prefers]
[DEBUG] >>>>>>> colder 3 teste.txt
prefers colder
[DEBUG] [colder] == [prefers colder]
[DEBUG] [colder] == [cat prefers colder]
[DEBUG] [colder] == [because the cat prefers colder]
[DEBUG] [colder] == [water]
because the cat prefers colder]
[DEBUG] [colder] = [warm water
because the cat prefers colder]
[DEBUG] [colder] == [drink warm water
because the cat prefers colder]
[DEBUG] [colder] == [like to drink warm water
because the cat prefers colder]
[DEBUG] [colder] = [doesn't like to drink warm water]
because the cat prefers colder]
[DEBUG] [colder] = [cat doesn't like to drink warm water]
because the cat prefers colder]
[DEBUG] [colder] == [owner
The cat doesn't like to drink warm water
because the cat prefers colder]
[DEBUG] [colder] = [its owner
The cat doesn't like to drink warm water
because the cat prefers colder]
[DEBUG] [colder] = [own's its owner
The cat doesn't like to drink warm water
because the cat prefers colder]
[DEBUG] [colder] = [cat own's it’s owner
1 Answers

The string pointed to by yytext can be used in a Flex action, but you need to be aware that the string is not persistent. As soon as yyflex is called again, the string will be modified; it is even possible that the memory it is stored in will be reallocated. In short, it's not your string; if you need it to stick around, you need to make your own copy. strdup is a good way of doing that (very soon, it will be part of standard C but it has been available for many years on almost all library implementations, since it's specified by Posix).

For reference, the documentation from the Flex manual:

When flex finds a match, yytext points to the first character of the match in the input buffer. The string itself is part of the input buffer, and is NOT allocated separately. The value of yytext will be overwritten the next time yylex() is called. In short, the value of yytext is only valid from within the matched rule’s action.

As that chapter goes on to note, if you make a copy of yytext, you should also make sure that the memory allocated for the copy is eventually free()d. Of course, that also applies to the other memory you malloc() in the course of your program.

Related