Parallelize the process to update the pandas column

Question

I have two data frames df_main & df_lookup.
df_main has 219k records with a geo-spatial feature POINT(latitude, longitude).
df_lookup has 12k records having geo-spatial feature LINESTRING(latitude longitude, ..........., latitude longitude)
Let me show you the dataframe for df_main.
df_lookup is also same, but instead of POINT it has LINESTRING

import pandas as pd
import numpy as np
df_main = pd.read_csv("lat_long.csv")
df_lookup = pd.read_csv("line.csv")

df_main

**Latitude  Longitude   point**
0   33.94   -80.37  POINT (33.9428727 -80.37280269999999)
1   42.71   -71.17  POINT (42.715921 -71.17759649999999)
2   40.75   -73.99  POINT (40.7513773 -73.9924884)
3   40.51   -78.40  POINT (40.51711 -78.402507)
4   33.04   -96.86  POINT (33.0458064 -96.86345449999999)

I am trying to get a minimum distance of a point in df_main to all the lines in df_lookup.
I am also updating the same distance in the column in df_main at each index iteration. Here is the code below :

cnt=0
for pnt in df_main.point:
    for line in df_lookup.line:
        min_dist = []
        min_dist.append(line.distance(pnt))
#         print("Result appended :", cnt)
    df_main.loc[cnt,'distance_GAF_locations'] = np.min(min_dist)
    df_main.to_csv("GAF_location_distance.csv", index = False)
    cnt+=1

This is taking way too much time.
Is there an easy way to Parallelize this process?

Parallelize the process to update the pandas column

0 Answers