Efficient filter, map, and reduce operation on a long list in Python

Benchmark with three new solutions (starting from a list of strings as the one you got from your file):

 4.1 s  original_numpy
11.6 s  original_listcomp
 4.8 s  better_listcomp
 1.7 s  even_better_listcomp
 0.6 s  counter

better_listcomp's only difference to yours is that I added .tolist(), so that price_list is actually a list and Python gets to work with Python ints instead NumPy ints, which is much faster. (Calling an array a list was really mislabeling).

even_better_listcomp goes a step further, never even going through NumPy at all. (And it multiplies with 1.05 just once, after adding.)

counter goes even further, delaying the conversion from string to int until after counting the price strings.

Code (Try it online!):

def better_listcomp(price_list):
    price_list = np.array(price_list).astype(int).tolist()
    return sum([x * 1.05 for x in price_list if x < 30])

def even_better_listcomp(price_list):
    prices = map(int, price_list)
    return sum([x for x in prices if x < 30]) * 1.05

def counter(price_list):
    return sum(int(k) * v for k, v in Counter(price_list).items() if int(k) < 30) * 1.05

def original_numpy(price_list):
    price_list = np.array(price_list).astype(int)
    return np.sum(price_list[price_list < 30] * 1.05)

def original_listcomp(price_list):
    price_list = np.array(price_list).astype(int)
    return sum([x * 1.05 for x in price_list if x < 30])

funcs = original_numpy, original_listcomp, better_listcomp, even_better_listcomp, counter

from time import time
import numpy as np
from collections import Counter

price_list = list(map(str, range(1, 100))) * 101010

for _ in range(2):
    for f in funcs:
        t = time()
        f(price_list)
        print(f'{time() - t :4.1f} s ', f.__name__)
    print()

Short answer: no.

Numpy uses vectorised version of math operations wherever it can. It means that if you have a nd.array of values, it can fit them into the SIMD registers of procesors. This means, that it can multiply tuples of numbers simultaneously (4, 8 or 16 at the same time depending on the SIMD version your processor supports.

Since Python is an interpreted language, you don't really have any alternatives. You would have to manually code procedures in a language that has access to the low level SIMD/AVX instructions of your processor and wrap it in the python API... which is exactly what NumPy is.

Answer

The Numpy is fast in the calculation but it is inefficient in converting the string list to an int list

Tips:

1. Use the Numpay array with NumPy functions only, using the Numpay data structure with pythonic functins is inefficient and vice-versa
2. Counter in some cases (when doing the conversion inside the counter) is the fastest among the Pythonic methods
3. To get the benefit of Numpy efficiency do the list conversion with python.

To better compare the results, the running time can be divided into three categories for this problem:

1. Time for reading from file (same for all tries)
2. Time to convert the string list into an int list (different for python list and numpy array)
3. Time to calculation (varies for different methods)

This separation is required to clearly show the efficiency of python list vs NumPy array. Having a look to the code:

import time
import numpy as np
from functools import reduce
from collections import Counter


def print_result(total_price, duration, method):
    print('{method:<35s} {0:>20.3f} (sec), total_price={1:<10.3f}'
          .format(duration, total_price, method=method))


start = time.time()
with open('price_list.txt') as f:
    price_list = f.read().split('\n')
print('Duration for reading from file: {0:<50.3f}'.format(time.time() - start))

start = time.time()
np_price_array = np.array(price_list).astype(int)  # uing numpy array
print('Duration for converting to numpy int array: {0:<50.3f}'.format(
    time.time() - start))

start = time.time()
price_list = list(map(int, price_list))
print('Duration for converting to python int list: {0:<50.3f}'.format(
    time.time() - start))

start = time.time()    
total_price = np.sum(np_price_array[np_price_array < 30] * 1.05)
print_result(total_price, time.time() - start, "NumPy function (numpy array)")

start = time.time()
total_price = sum([x for x in np_price_array if x < 30])*1.05
print_result(total_price, time.time() - start,
             "List comprehension (numpy array)")


start = time.time()
total_price = sum((x for x in price_list if x < 30))*1.05
print_result(total_price, time.time() - start, "List generator (Python list)")

start = time.time()
total_price = sum([x for x in price_list if x < 30])*1.05
print_result(total_price, time.time() - start,
             "List comprehension (Python list)")

start = time.time()
total_price = sum([float(x[0]) * float(x[1])
                  for x in Counter(price_list).items() if x[0] < 30]) * 1.05
print_result(total_price, time.time() - start, "Counter (Python list)")

start = time.time()
total_price = reduce(
    lambda x, y: x+y, filter(lambda x: x < 30, price_list)) * 1.05
print_result(total_price, time.time() - start, "functions (Python list)")

Following chart is as the result of the code output:

Numpy is written in C with efficient routines specialized for vectorial operations on numeric data.

Another particularity is that the data is stored in specialized containers (arrays) that optimize memory access.

In your case you could use:

import numpy as np

# load the data directly as numpy array
a = np.loadtxt('price_list.txt', dtype=np.int64)

# filter, sum, multiply
out = (a[a<30]).sum()*1.05