Removing all zeros from an array

Viewed 5513

I have an array of shape [120000, 3] in which only the first 1500 elements are useful and the others are 0.

Here an example

[15.0, 14.0, 13.0]
[11.0, 7.0, 8.0]
[4.0, 1.0, 3.0]
[0.0, 0.0, 0.0]
[0.0, 0.0, 0.0]
[0.0, 0.0, 0.0]
[0.0, 0.0, 0.0]

I have to find a way to remove all the elements that are [0.0, 0.0, 0.0]. I tried to write this but it doesn't work

for point in points:
        if point[0] == 0.0 and point[1] == 0.0 and point[2] == 0.0:
            np.delete(points, point)

edit

All the solutions in the comment work, but I gave the green tick to the one I have used. Thanks to all.

6 Answers

Don't use for loops--those are slow. Calling np.delete repeatedly in a for loop is a recipe for poor performance.

Instead, create a mask:

zero_rows = (points == 0).all(1)

That is an array of length 120000 which is True where all the elements in that row are 0.

Then find the first such row:

first_invalid = np.where(zero_rows)[0][0]

Finally, slice the array:

points[:first_invalid]

There are a few related approaches, split into two camps. You can either use a vectorised approach via calculation of a single Boolean array and np.ndarray.all. Or you can calculate the index of the first row which contains only 0 elements, either via a for loop or next with a generator expression.

For performance, I recommend you use numba with a manual for loop. Here's one example, but see benchmarking below for a more efficient variant:

from numba import jit

@jit(nopython=True)
def trim_enum_nb(A):
    for idx in range(A.shape[0]):
        if (A[idx]==0).all():
            break
    return A[:idx]

Performance benchmarking

# python 3.6.5, numpy 1.14.3

%timeit trim_enum_loop(A)     # 9.09 ms
%timeit trim_enum_nb(A)       # 193 µs
%timeit trim_enum_nb2(A)      # 2.2 µs
%timeit trim_enum_gen(A)      # 8.89 ms
%timeit trim_vect(A)          # 3.09 ms
%timeit trim_searchsorted(A)  # 7.67 µs

Test code

Setup

import numpy as np
from numba import jit

np.random.seed(0)

n = 120000
k = 1500

A = np.random.randint(1, 10, (n, 3))
A[k:, :] = 0

Functions

def trim_enum_loop(A):
    for idx, row in enumerate(A):
        if (row==0).all():
            break
    return A[:idx]

@jit(nopython=True)
def trim_enum_nb(A):
    for idx in range(A.shape[0]):
        if (A[idx]==0).all():
            break
    return A[:idx]

@jit(nopython=True)
def trim_enum_nb2(A):
    for idx in range(A.shape[0]):
        res = False
        for col in range(A.shape[1]):
            res |= A[idx, col]
            if res:
                break
            return A[:idx]

def trim_enum_gen(A):
    idx = next(idx for idx, row in enumerate(A) if (row==0).all())
    return A[:idx]

def trim_vect(A):
    idx = np.where((A == 0).all(1))[0][0]
    return A[:idx]

def trim_searchsorted(A):
    B = np.frombuffer(A, 'S12')
    idx = A.shape[0] - np.searchsorted(B[::-1], B[-1:], 'right')[0]
    return A[:idx]

Checks

# check all results are the same
assert (trim_vect(A) == trim_enum_loop(A)).all()
assert (trim_vect(A) == trim_enum_nb(A)).all()
assert (trim_vect(A) == trim_enum_nb2(A)).all()
assert (trim_vect(A) == trim_enum_gen(A)).all()
assert (trim_vect(A) == trim_searchsorted(A)).all()
x = [[15.0, 14.0, 13.0],
[11.0, 7.0, 8.0],
[4.0, 1.0, 3.0],
[0.0, 0.0, 0.0],
[0.0, 0.0, 0.0],
[0.0, 0.0, 0.0],
[0.0, 0.0, 0.0]]

The simple iterative solution:

y = [i for i in x if i != [0.0, 0.0, 0.0]]

A better solution (Python 3.x):

y = list(filter(lambda a: a != [0.0, 0.0, 0.0], x))

output:

[[15.0, 14.0, 13.0], [11.0, 7.0, 8.0], [4.0, 1.0, 3.0]]

know it is over just thought i would give my answer :)

x = [[15.0, 14.0, 13.0],
[11.0, 7.0, 8.0],
[4.0, 1.0, 3.0],
[0.0, 0.0, 0.0],
[0.0, 0.0, 0.0],
[0.0, 0.0, 0.0],
[0.0, 0.0, 0.0]]

then a simeple list comprehension can be made

[i for i in x if all(i)]

and outputs:

[[15.0, 14.0, 13.0],[11.0, 7.0, 8.0],[4.0, 1.0, 3.0]]

takes

0.0000010866 # seconds or 1.0866 microseconds

take the time with a gram of salt it is really inconsisten give my 2 sec to get a better estimation

when:

x = [[15.0, 14.0, 13.0],
[11.0, 7.0, 8.0],
[4.0, 1.0, 3.0],
[0.0, 0.0, 0.0],
[0.0, 0.0, 0.0],
[0.0, 0.0, 0.0],
[0.0, 0.0, 0.0]]*(120000//7)

i get the time to

0.01199 # seconds

this time is hugely based on wather they are 0 or not, 0 are way faster since it is ignored.

For logarithmic complexity, you can use numpy.searchsorted after casting data by rows:

B=np.frombuffer(A,'S12')
index=B.size-np.searchsorted(B[::-1],B[-1:],'right')[0]

index will be the number of not null items if the first ones are all not null.

test :

>>>> %timeit B.size-searchsorted(B[::-1],B[-1:],'right')[0]
2.2 µs 

A simple iterative solution with usage of vstack

import numpy as np
b = np.empty((0,3), float)
for elem in a:
    toRemove = np.array([0.0, 0.0, 0.0])
    if(not np.array_equal(elem,toRemove)):
        b=np.vstack((b, elem))
Related