Efficient way to remove a list of strings from a 2D array (list of lists) in Python?

user_15175417 0
Viewed 106

I have quite a large dataset in the form of a list of lists. Basically, these list of lists are tokenized sentences (Each row is a sentence and each sentence is tokenized).

train_dataset - The 2D array (or list of lists) contain around 150000 sentences/rows and each sentence roughly contains 12 to 15 words.

td - Another 1D list contains 55000 UNIQUE words. If any word from these words are present in the 2D array, I need to replace them with .

I tried below ways to do so but they take a very long time, even in Google Colab's different runtime environments:

train_dataset = ['<UNK>' if train_dataset[i][j] in td else train_dataset[i][j] for i in range(0,len(train_dataset)) for j in range(0,len(train_dataset[i])) ]
train_dataset

I also tried to convert the list to NUMPY arrays and try so. Still no luck.

import numpy as np
np_tr = np.array(train_dataset)
for st in td:
  for i in range(0,len(np_tr)):
    for j in range(0,len(np_tr[i])):
      if np_tr[i][j] == st:
        np_tr[i][j] = '<UNK>'

I want to know if there is any efficient way to do this in Python. I don't have access to PySpark or Hadoop to achieve this.

Thanks in advance!

multidimensional-array list python
1 Answers

Try converting the 1D list to a set. That should make lookup O(1) as opposed to O(len(td)). Then you should be able to do what you were doing before and it should be quite a bit faster I think. Keep in mind that what you were doing was actually flattening the 2D list into a 1D list. I'm not sure if that is what you want.

td = set(td)
train_dataset = ['<UNK>' if train_dataset[i][j] in td else train_dataset[i][j] for i in range(0,len(train_dataset)) for j in range(0,len(train_dataset[i]))]
train_dataset
user_99926140
Related