Matching arbitrary number of following tokens in stream using functional style

The problem is as follows:

1. there is a file with tokens - each token is in a separate line accompanied by some meta-data (e.g. document id),
2. some sequences of the tokens should be counted, a sequence might be one or more tokens,
3. the sequence are kept in a trie, but this is not a requirement,
4. the implementation has to be very efficient, since the file to be processed has gigabytes of data.

My present implementation (in Ruby) is as follows:

def convert_tuple(tuple)
  document_id, token_index, space, token = *tuple
  token = token.chomp
  token.force_encoding("ascii-8bit")
  document_id = document_id.to_i
  [document_id, token_index, space, token]
end

def count_and_match_tokens(string, index, counts, document_id, first_token_index, last_token_index)
  token_id = index[string]
  if token_id
    STDERR.puts "%s\t%s\t%s\t%s" % [document_id, first_token_index, last_token_index, string]
    counts[string] += 1
  end
  index.search(string).size > 0
end

counts = Hash.new(0)
index = Melisa::IntTrie.new
index.load(index_path)

CSV.open(input_path, col_sep: "\t") do |input|
  input.each do |tuple|
    document_id, first_token_index, space, token = convert_tuple(tuple)
    recoreded_pos = input.pos
    last_token_index = first_token_index
    string = token.dup
    while(count_and_match_tokens(string, index, counts, document_id, first_token_index, last_token_index)) do
      last_document_id, last_token_index, space, last_token = convert_tuple(input.shift)
      break if document_id != last_document_id
      string << " " if space == "1"
      string << last_token
    end
    input.pos = recoreded_pos
  end
end  

CSV.open(output_path,"w") do |output|
  counts.each do |tuple|
    output << tuple
  end
end

The convert_tuple function only makes basic conversion of the data (i.e. converts strings to numbers, etc.).

The count_and_match_tokens function counts the tokens and returns true, if the passed string argument is a prefix of a different string. I use a trie structure to effectively verify this condition.

I am wondering how would look a solution that is written using functional style. The problem I am facing is the fact that the matched sequence might span many tokens.

In Ruby (or in OO style in general) I can record the position where I started the matching (recorded_pos = input.pos) and "reset" the stream, when the sub-sequence matching is over (input.pos = recorded_pos). As a result a subsequent call to each will return the next token that is in the stream. So tokens inside already recognized sequences (the tokens that are processed inside the while loop) can also be first matching tokens in other sub-sequences.

I would be thankful for a solution in Elixir, but any other functional language would be OK as well.

EDIT

I have provided the definitions of convert_tuple and count_and_match_tokens as well as example input and output (the files are truncated, thus the counts does not directly correspond to the input file).

The index data structure that appears in the code is a Maris Trie (Melisa gem: https://github.com/wordtreefoundation/melisa/ )

Example input:

0   746 1   The
0   748 1   river
0   751 1   Bosna
0   754 1   (
0   763 0   )
0   765 1   (
0   766 0   Cyrillic
0   767 0   :
0   769 1   Босна
0   770 0   )
0   772 1   is
0   774 1   the
0   776 1   third
0   778 1   longest
0   781 1   river
0   784 1   in
0   787 1   Bosnia
0   789 1   and
0   791 1   Herzegovina
0   793 0   ,
0   795 1   and
0   797 1   is
0   799 1   considered
0   801 1   one
0   803 1   of
0   805 1   the
0   807 1   country
0   808 0   '
0   809 0   s
0   811 1   three
0   813 1   major
0   815 1   internal
0   817 1   rivers

Token sequence to be recognized:

Bosnia
Bosnia and Herzegovina
river
Herzegovina

Example output:

river,2
Bosnia,1
Bosnia and Herzegovina,1
Herzegovina,1

I hope this helps in understanding the problem I am trying to solve.