Why is full_like so slow (compared to other approaches)?

user_4533188 0
Viewed 140

When I was refactoring my code, I noticed that I wrote a line

    class_proba = np.empty_like(self.classes_, dtype=np.float64)
    class_proba[:] = np.NaN

so I thought, it would be faster to use full_like. But since the first rule of performance is "measure everything" (and not just "don't talk about performance") - I did that, and was quite surprised:

import numpy as np
import time

t0 = time.time()
for _ in range(500):
   np.full_like(np.array([1,2,3]), np.nan, dtype=np.float64)
print(time.time()-t0)

t0 = time.time()
for _ in range(500):
   class_proba = np.empty_like(np.array([1,2,3]), dtype=np.float64)
   class_proba[:] = np.NaN
print(time.time()-t0)

results in

0.008994579315185547
0.0019996166229248047

I was very surprised. Maybe it was the usage of "empty" that made the first part so fast or the wrong type? But even

t0 = time.time()
for _ in range(500):
    class_proba = np.ones_like(np.array([1, 2, 3]), dtype=np.float64)
    class_proba[:] = np.NaN
print(time.time() - t0)

is faster than using full_like with

0.003997087478637695

and

t0 = time.time()
for _ in range(500):
   np.full_like(np.array([1,2,3]), np.nan, dtype=float)
print(time.time()-t0)

is (better but) still not as fast with

0.0029981136322021484

So my questions is: Why is it so slow to use full_like in this case - compared to the other two approaches?

Next I tried to make the array much larger:

    a = np.arange(1000000)

    t0 = time.time()
    for _ in range(500):
        np.full_like(a, np.nan, dtype=np.float64)
    print(time.time() - t0)

    t0 = time.time()
    for _ in range(500):
        np.full_like(a, np.nan, dtype=float)
    print(time.time() - t0)

    t0 = time.time()
    for _ in range(500):
        class_proba = np.empty_like(a, dtype=np.float64)
        class_proba[:] = np.NaN
    print(time.time() - t0)

    t0 = time.time()
    for _ in range(500):
        class_proba = np.ones_like(a, dtype=np.float64)
        class_proba[:] = np.NaN
    print(time.time() - t0)

resulting in

1.9398648738861084
1.819936990737915
1.853914499282837
2.292659044265747

Can someone explain this as well?

numpy python performance
1 Answers

If the timeit module from the Python standard library is used, the performance does not match what was reported in the OP's question. np.full_like and np.empty_like ... out[:]=nan perform similarly, and np.ones_like ... out[:]=nan was slowest.

This makes intuitive sense when considering the number of operations. All of the methods here use np.empty_like at their core. np.ones_like creates an empty array and then overwrites each value with 1. np.full_like creates an empty array and then overwrites each value with whatever was specified in the np.full_like call. It makes sense that np.ones_like...out[:]=np.nan is the slowest, because it creates an empty array, fills it with 1, then fills it with np.nan. On the other hand, np.full_like and np.empty_like...out[:]=np.nan are similar in terms of number of operations, and have almost identical performance.

import timeit

timeit.main(
    ["--setup", "import numpy as np; a = np.arange(1000000)",
    "class_proba = np.full_like(a, np.nan, dtype=np.float64)"])

timeit.main(
    ["--setup", "import numpy as np; a = np.arange(1000000)",
    """\
class_proba = np.empty_like(a, dtype=np.float64)
class_proba[:] = np.NaN"""])

timeit.main(
    ["--setup", "import numpy as np; a = np.arange(1000000)",
    """\
class_proba = np.ones_like(a, dtype=np.float64)
class_proba[:] = np.NaN"""])

The output is below:

1000 loops, best of 3: 616 usec per loop
1000 loops, best of 3: 619 usec per loop
1000 loops, best of 3: 1.23 msec per loop
user_56660870
Related