I have written a MWE to try to summarize my problem. If I have 8 NumPy arrays and I want to filter each of them based on conditions related to all of the arrays, what is a more efficient way to achieve this (with respect to memory/speed)?
import numpy as np
import pandas as pd
import time
# data generation
data = dict()
for key in ["a","b","c","d","e","f","g","h"]:
data[key] = np.random.rand(1000000)
Attempt #1:
start = time.perf_counter()
validmask = (data["a"] > 0.2) & (data["b"] < 0.8) & (data["c"] > 0.5) & (data["d"] > 0.9) & (data["e"] < 0.9) & (data["f"] < 0.8) & (data["g"] > 0.1) & (data["h"] > 0.5)
validdata = dict()
for key in data.keys():
validdata[key] = data[key][validmask]
print(f"time = {time.perf_counter()-start}")
Yields time = 0.028449237000018.
Attempt #2:
df = pd.DataFrame()
for key in data.keys():
df[key] = data[key]
start = time.perf_counter()
df[(df["a"] > 0.2) & (df["b"] < 0.8) & (df["c"] > 0.5) & (df["d"] > 0.9) & (df["e"] < 0.9) & (df["f"] < 0.8) & (df["g"] > 0.1) & (df["h"] > 0.5)]
print(f"time = {time.perf_counter()-start}")
Yields time = 0.08374083499995777.