Keras Tokenizer only tokenizing the first row of CSV file

user_17110025 0
Viewed 132

I'm really new to the keras API, and I may be stuck on a very simple task. I have a csv file with 4 columns. Currently I only want to use 1 of these columns. I'm using the pandas library to read the csv and have selected to only use the column 'host'.

This works ok, however when I do the process of tokenizing the data through the keras tokenizer function, it only reads the first row in the csv file.

I need the tokenizer to read the csv and tokenize it at a character level, which it appears to be doing but only for the first row. Please see code below and any help is much appreciated.

fields=['host']
test_dataset = pd.read_csv('dga_data.csv',usecols=fields)

test_dataset_tok= Tokenizer(split=',',char_level=True, oov_token=True)
print(test_dataset_tok)

test_dataset_tok.fit_on_texts(test_dataset)
print(test_dataset_tok)

test_dataset_sequences=test_dataset_tok.texts_to_sequences(test_dataset)
print(test_dataset_sequences)
print(test_dataset_tok.word_index)
pandas keras csv python
1 Answers

You are passing a Dataframe to fit_on_texts and you need to pass a list. From documentation:

texts : can be a list of strings, a generator of strings (for memory-efficiency), or a list of list of strings.

Therefore, you need to pass a list, or at least a Pandas Series, so when the fit_on_texts executes this for loop it iterates over each row of CSV file, and not just the dataframe axis label.

In [22]: type(test_dataset)
Out[22]: pandas.core.frame.DataFrame

In [23]: type(test_dataset['host'])
Out[23]: pandas.core.series.Series

import pandas as pd
from tensorflow.keras.preprocessing.text import Tokenizer

test_dataset = pd.DataFrame({'host': [
    'Aspire to inspire before we expire.',
    'Let the beauty of what you love be what you do.',
    'The meaning of life is to give life meaning.',
    'I have nothing to lose but something to gain.',
]})

# pandas.core.series.Series
test_dataset = test_dataset['host']

test_dataset_tok= Tokenizer(split=',',char_level=True, oov_token=True)
print(test_dataset_tok)

test_dataset_tok.fit_on_texts(test_dataset)
print(test_dataset_tok)

test_dataset_sequences=test_dataset_tok.texts_to_sequences(test_dataset)
print(test_dataset_sequences)
print(test_dataset_tok.word_index)

Output:

<keras_preprocessing.text.Tokenizer object at 0x0000019AFFA65CD0>
<keras_preprocessing.text.Tokenizer object at 0x0000019AFFA65CD0>
[
    [8, 11, 18, 4, 14, 3, 2, 6, 5, 2, 4, 7, 11, 18, 4, 14, 3, 2, 15, 3, 12, 5, 14, 3, 2, 19, 3, 2, 3, 23, 18, 4, 14, 3, 16],
    [13, 3, 6, 2, 6, 9, 3, 2, 15, 3, 8, 17, 6, 20, 2, 5, 12, 2, 19, 9, 8, 6, 2, 20, 5, 17, 2, 13, 5, 21, 3, 2, 15, 3, 2, 19, 9, 8, 6, 2, 20, 5, 17, 2, 24, 5, 16],
    [6, 9, 3, 2, 22, 3, 8, 7, 4, 7, 10, 2, 5, 12, 2, 13, 4, 12, 3, 2, 4, 11, 2, 6, 5, 2, 10, 4, 21, 3, 2, 13, 4, 12, 3, 2, 22, 3, 8, 7, 4, 7, 10, 16], 
    [4, 2, 9, 8, 21, 3, 2, 7, 5, 6, 9, 4, 7, 10, 2, 6, 5, 2, 13, 5, 11, 3, 2, 15, 17, 6, 2, 11, 5, 22, 3, 6, 9, 4, 7, 10, 2, 6, 5, 2, 10, 8, 4, 7, 16]
]
{
    True: 1, ' ': 2, 'e': 3, 'i': 4, 'o': 5, 't': 6, 'n': 7, 'a': 8,
    'h': 9, 'g': 10, 's': 11, 'f': 12, 'l': 13, 'r': 14, 'b': 15, '.': 16,
     'u': 17, 'p': 18, 'w': 19, 'y': 20, 'v': 21, 'm': 22, 'x': 23, 'd': 24
}
user_131441430
Related