I am just starting to explore Bert in a multiclass text classification task. In doing that, I am using this data. This is my code. During model testing, attempt to compute cosine similarities:
#--- Model Algorithm ---#
## compute cosine similarities
similarities = np.array(
[metrics.pairwise.cosine_similarity(X, y).T.tolist()[0]
for y in dic_y.values()]
).T
gives this error:
ValueError: Input contains NaN, infinity or a value too large for dtype('float32').
From many stackoverflow posts, there were suggestions to eliminate NaN and null cells which I did at the beginning (see my code). There were also suggestions to replace NaN but I when I did as thus:
X = np.nan_to_num(X.astype(np.float32))
similarities = np.array(
[metrics.pairwise.cosine_similarity(X, y).T.tolist()[0]
for y in dic_y.values()]
).T
I got another error:
ValueError: Expected 2D array, got 1D array instead:
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.
Applying .reshape(-1, 1) or .reshape(1, -1) to X:
X = np.reshape(X, (-1, 1))
similarities = np.array(
[metrics.pairwise.cosine_similarity(X, y).T.tolist()[0]
for y in dic_y.values()]
).T
generates the same error:
ValueError: Expected 2D array, got 1D array instead:
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.
I am sure that the input X is a 2D array and not 1D, because X.ndim is 2. Any help will be appreciated.