I am running my first cnn text-classifier using the IMDB dataset with the in-built
tf.keras.datasets.imdb.load_data()
I understand the AttributeError: 'int' object has no attribute 'lower' error indicates that a lowercase function is being applied to int objects (seems to be from the tokenizer). However, I don't know why it is throwing this in this case as I am loading it directly though the in-built tf.keras.datasets.imdb.load_data().
I am not experienced with using embedding in text-classification.
The code excluding the CNN model is:
import tensorflow as tf
from keras.preprocessing import sequence
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation
from keras.layers import Embedding, LSTM
from keras.layers import Conv1D, Flatten, MaxPooling1D
from keras.datasets import imdb
import wandb
from wandb.keras import WandbCallback
import numpy as np
from keras.preprocessing import text
import imdb
wandb.init(mode="disabled") # disabled for debugging
config = wandb.config
# set parameters:
config.vocab_size = 1000
config.maxlen = 1000
config.batch_size = 32
config.embedding_dims = 10
config.filters = 16
config.kernel_size = 3
config.hidden_dims = 250
config.epochs = 10
(X_train, y_train), (X_test, y_test) = tf.keras.datasets.imdb.load_data()
tokenizer = text.Tokenizer(num_words=config.vocab_size)
tokenizer.fit_on_texts(X_train)
X_train = tokenizer.texts_to_matrix(X_train)
X_test = tokenizer.texts_to_matrix(X_test)
X_train = sequence.pad_sequences(X_train, maxlen=config.maxlen)
X_test = sequence.pad_sequences(X_test, maxlen=config.maxlen)
Line 34 referred to in the error is tokenizer = text.Tokenizer(num_words=config.vocab_size)
The exact error thrown (includes Deprecation warnings) is:
C:\Users\Keegan\anaconda3\envs\oldK\lib\site-
packages\tensorflow_core\python\keras\datasets\imdb.py:129:
VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-
or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If
you meant to do this, you must specify 'dtype=object' when creating the ndarray.
x_train, y_train = np.array(xs[:idx]), np.array(labels[:idx])
C:\Users\Keegan\anaconda3\envs\oldK\lib\site-
packages\tensorflow_core\python\keras\datasets\imdb.py:130: VisibleDeprecationWarning: Creating
an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or
ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must
specify 'dtype=object' when creating the ndarray.
x_test, y_test = np.array(xs[idx:]), np.array(labels[idx:])
Traceback (most recent call last):
File "imdb-cnn.py", line 34, in <module>
tokenizer.fit_on_texts(X_train)
File "C:\Users\Keegan\anaconda3\envs\oldK\lib\site-packages\keras_preprocessing\text.py",
line 217, in fit_on_texts
text = [text_elem.lower() for text_elem in text]
File "C:\Users\Keegan\anaconda3\envs\oldK\lib\site-packages\keras_preprocessing\text.py", line 217, in <listcomp>
text = [text_elem.lower() for text_elem in text]
AttributeError: 'int' object has no attribute 'lower'
The Anaconda venv has Python 3.7.1, Tensorflow 2.1.0 and Keras 2.3.1