Python - For loop millions of rows

Your question was if there is a way to vectorize the for loop, but I think that question hides what you really want which is an easy way to speed your code up. For performance questions, a good starting point is always profiling. However, I have a strong suspicion that the dominant operation in your code is .query(row.query_string). Running that for every row is expensive if arr is large.

For arbitrary queries, that runtime can't really be improved at all without removing dependencies between iterations and parallelizing the expensive step. You might be a bit luckier though. Your query string always checks two different columns to see if they're equal to something you care about. However, for each row that requires going through your entire slice of arr. Since the slice changes each time, that could cause problems, but here are some ideas:

• Since you're slicing arr each time anyway, maintain a view of just the arr.Already_Linked==0 rows so you're iterating over a smaller object.
• Better yet, before you do any looping you should first group arr by Terminal and Operator. Then, instead of running through all of arr, first select the group you want and then do your slicing and filtering. This would require rethinking the exact implementation of query_string a little bit, but the advantage is that if you have a lot of terminals and operators you'll typically be working over a much smaller object than arr. Moreover, you wouldn't even have to query that object since that was implicitly done by the groupby.
• Depending on how aux.hour typically relates to row.hour_aux, you might have improvements by sorting aux at the beginning with respect to hour. Just using the inequality operator you probably wouldn't see any gains, but you could pair that with a logarithmic search for the cutoff point and then just slice up to that cutoff point.
• And so on. Again, I suspect any way of restructuring the query you're doing on all of arr for every row will offer substantially more gains than just switching frameworks or vectorizing bits and pieces.

Expanding on some of those points a little bit and adapting @DJK's code a bit, look at what happens when we have the following changes.

groups = arr.groupby(['Operator', 'Terminal'])

for row in c[(c.A_D == 'D') & (c.Already_linked == 0)].itertuples():
    g = groups.get_group((row.Operator, row.Terminal))
    vb = g[(g.Already_linked==0) & (g.hour<row.hour_aux)]
    try:
        aux = (vb.START - row.x).abs().idxmin()
        print(row.x)
        c.loc[row.Index, 'a'] = vb.loc[aux,'FlightID']
        g.loc[aux, 'Already_linked'] = 1
        continue
    except:
        continue

Part of the reason your query is so slow is because it's searching over all of arr each time. In contrast, the .groupby() executes in roughly the same time as one query, but then for every subsequent iteration you can just use .get_group() to efficiently find the tiny subset of the data you care about.

A helpful (extremely crude) rule of thumb when benchmarking is that a billion things takes a second. If you're seeing much longer times than that for something measured in millions of things, like your millions of rows, that means that for each of those rows you're doing tons of things to get up to billions of operations. That leaves a ton of potential for better algorithms to reduce the number of operations, whereas vectorization really only yields constant factor improvements (and for many string/query operations not even great improvements at that).

While this is not a vecterized solution, it should speed things up rapidly if your sample data set mimics your true data set. Currently, you are wasting time looping over every row, but you only care about looping over rows where ['A_D'] == 'D' and ['Already_linked'] ==0. Instead remove the if's and loop over the truncated dataframe which is only 30% of the initial dataframe

for row in c[(c.A_D == 'D') & (c.Already_linked == 0)].itertuples():

    vb = arr[(arr.Already_linked == 0) & (arr.hour < row.hour_aux)].copy().query(row.query_string)
    try:
        aux = (vb.START - row.x).abs().idxmin()
        print(row.x)
        c.loc[row.Index, 'a'] = vb.loc[aux,'FlightID']
        arr.loc[aux, 'Already_linked'] = 1
        continue
    except:
        continue

This solution uses pd.DataFrame.isin which uses numpy.in1d

Apparently 'isin' isn't necessarily faster for small datasets (like this sample), but is significantly faster for large datasets. You'll have to run it against your data to determine performance.

flight_record_linkage.ipynb

Expanded the dataset using c = pd.concat([c] * 10000, ignore_index=True)

• Increase the dataset length by 3 orders of magnitude (10000 rows total).
  • Original method: Wall time: 8.98s
  • New method: Wall time: 16.4s
• Increase the dataset length by 4 orders of magnitude (100000 rows total).
  • Original method: Wall time: 8min 17s
  • New method: Wall time: 1min 14s
• Increase the dataset length by 5 orders of magnitude (1000000 rows total).
  • New method: Wall time: 11min 33s

New Method: Using isin and apply

def apply_do_g(it_row):
    """
    This is your function, but using isin and apply
    """

    keep = {'Operator': [it_row.Operator], 'Terminal': [it_row.Terminal]}  # dict for isin combined mask

    holder1 = arr[list(keep)].isin(keep).all(axis=1)  # create boolean mask
    holder2 = arr.Already_linked.isin([0])  # create boolean mask
    holder3 = arr.hour < it_row.hour_aux  # create boolean mask

    holder = holder1 & holder2 & holder3  # combine the masks

    holder = arr.loc[holder]

    if not holder.empty:

        aux = np.absolute(holder.START - it_row.x).idxmin()

        c.loc[it_row.name, 'a'] = holder.loc[aux].FlightID  # use with apply 'it_row.name'

        arr.loc[aux, 'Already_linked'] = 1


def new_way_2():
    keep = {'A_D': ['D'], 'Already_linked': [0]}
    df_test = c[c[list(keep)].isin(keep).all(axis=1)].copy()  # returns the resultant df
    df_test.apply(lambda row: apply_do_g(row), axis=1)  # g is multiple DataFrames"


#call the function
new_way_2()

Your problem looks like one of the most common problems in database operation. I do not fully understand what you want to get because you have not formulated the task. Now to the possible solution - avoid loops at all.

You have a very long table with columns time, FlightID, Operator, Terminal, A_D. Other columns and dates do not matter if I understand you correctly. Also start_time and end_time are the same in every row. By the way you may get time column with the code table.loc[:, 'time'] = table.loc[:, 'START'].dt.time.

1. table = table.drop_duplicates(subset=['time', 'FlightID', 'Operator', 'Terminal']). And your table gonna become significantly shorter.

2. Split table into table_arr and table_dep according to A_D value: table_arr = table.loc[table.loc[:, 'A_D'] == 'A', ['FlightID', 'Operator', 'Terminal', 'time']], table_dep = table.loc[table.loc[:, 'A_D'] == 'D', ['FlightID', 'Operator', 'Terminal', 'time']]

3. Seems like all you tried to get with loops you may get with a single line: table_result = table_arr.merge(table_dep, how='right', on=['Operator', 'Terminal'], suffixes=('_arr', '_dep')). It is basically the same operation as JOIN in SQL.

According to my understanding of your problem and having the tiny piece of data you have provided you get just the desired output (correspondence between FlightID_dep and FlightID_arr for all FlightID_dep values) without any loop so much faster. table_result is:

  FlightID_arr Operator  Terminal  time_arr FlightID_dep  time_dep
0        DL401       DL         3  06:50:00        DL001  09:30:00
1        DL402       DL         3  07:45:00        DL001  09:30:00
2          NaN       VS         3       NaN        VS001  15:15:00

Of course, in general case (with actual data) you will need one more step - filter table_result on condition time_arr < time_dep or any other condition you have. Unfortunately the data you have provided is not enough to fully solve your problem.

Complete code is:

import io
import pandas as pd

data = '''
START,END,A_D,Operator,FlightID,Terminal,TROUND_ID,tot
2017-03-26 16:55:00,2017-10-28 16:55:00,A,QR,QR001,4,QR002,70
2017-03-26 09:30:00,2017-06-11 09:30:00,D,DL,DL001,3,,84
2017-03-27 09:30:00,2017-10-28 09:30:00,D,DL,DL001,3,,78
2017-10-08 15:15:00,2017-10-22 15:15:00,D,VS,VS001,3,,45
2017-03-26 06:50:00,2017-06-11 06:50:00,A,DL,DL401,3,,9
2017-03-27 06:50:00,2017-10-28 06:50:00,A,DL,DL401,3,,19
2017-03-29 06:50:00,2017-04-19 06:50:00,A,DL,DL401,3,,3
2017-05-03 06:50:00,2017-10-25 06:50:00,A,DL,DL401,3,,32
2017-06-25 06:50:00,2017-10-22 06:50:00,A,DL,DL401,3,,95
2017-03-26 07:45:00,2017-10-28 07:45:00,A,DL,DL402,3,,58
'''

table = pd.read_csv(io.StringIO(data), parse_dates=[0, 1])
table.loc[:, 'time'] = table.loc[:, 'START'].dt.time
table = table.drop_duplicates(subset=['time', 'FlightID', 'Operator', 'Terminal'])
table_arr = table.loc[table.loc[:, 'A_D'] == 'A', ['FlightID', 'Operator', 'Terminal', 'time']]
table_dep = table.loc[table.loc[:, 'A_D'] == 'D', ['FlightID', 'Operator', 'Terminal', 'time']]

table_result = table_arr.merge(
    table_dep,
    how='right',
    on=['Operator', 'Terminal'],
    suffixes=('_arr', '_dep'))
print(table_result)