My question might sound a little confusing but what I am trying to do is find an efficient method to iterate over a list that contains arrays and remove arrays from the list if they have the same entries at certain positions. Let's say I have a list with 3x2 arrays and want to make the list unique regarding the last two elements in the bottom row for example. What I came up with so far is the following:
import numpy as np
my_array_list = [np.array([[1,2,3],[4,5,6]]), np.array([[9,8,7],[6,5,4]]),
np.array([[2,3,4],[5,6,7]]), np.array([[1,7,8],[0,5,6]])]
while i < len(my_array_list):
j = i + 1
while j < len(my_array_list):
if my_array_list[i][1,1] == my_array_list[j][1,1] and my_array_list[i][1,2] == my_array_list[j][1,2]:
del my_array_list[j]
else:
j += 1
i += 1
print(my_array_list)
>>> my_array_list = [np.array([[1,2,3],[4,5,6]]), np.array([[9,8,7],[6,5,4]]),
np.array([[2,3,4],[5,6,7]])] ## Since 5,6 is already in the last two colums of the first array the last array got deleted
This loop does what I want but the problem is that it is very slow. The data I am having will be generated from a Monte Carlo Simulation thus there will most likely be millions of arrays in my list. I was wondering if there was a faster way to do this like to somehow remember which combinations have already been encountered so I only have to loop over the list once and not len(my_array_list)-times for every single combination.
Thanks in advance, any help would be appreciated. Cheers!