Getting Country Name from coordinates takes too Long

user_14534902 0
Viewed 247

I have a data from NASA websites which has bunch of coordinates and from these coordinates, I'm trying to find which country they belong via geopy in every 1 hour. But unfortunately it takes quite a lot to get the country names .

How can I reduce the process of time ?

Here's my code;

import pandas as pd
import numpy as np
from geopy.geocoders import Nominatim
geolocator = Nominatim(user_agent="geoapiExercises")

df=pd.read_csv("https://firms.modaps.eosdis.nasa.gov/data/active_fire/modis-c6.1/csv/MODIS_C6_1_Europe_24h.csv")
df.latitude=df.latitude.astype(str)
df.longitude=df.longitude.astype(str)


country_list=[]

for i in range(len(df)):
    try:
        location = geolocator.reverse(df.latitude[i]+","+df.latitude[i])
        country=location.raw["address"]["country"]
        country_list.append(country)
    except:
        country_list.append("None")
        print(i)
        continue

df_s=pd.Series(country_list)
df["Country"]=df_s
df.to_csv("map_data_R00.csv",index=False)
pandas dataframe python
1 Answers

It is "slow" because, as it says in the Nominatim Usage Policy:

No heavy uses (an absolute maximum of 1 request per second).

So at least it will take about len(df) seconds, as I think they apply throttling. Also, you might have been blocked:

Clients sending repeatedly the same query may be classified as faulty and blocked.

If you want to speed it up, you can run your own instance of the service. The docs contain instructions on how to install the software and import their data.

That said, your code contains a bug: you are using the latitude twice. Also, iterating over range(len(...)) is a code smell in Python. I would go for something like

import pandas as pd
from geopy.geocoders import Nominatim
geolocator = Nominatim(
    user_agent="StackOverflow Question (https://stackoverflow.com/a/68727013/383793)"
)

df = pd.read_csv("https://firms.modaps.eosdis.nasa.gov/data/active_fire/modis-c6.1/csv/MODIS_C6_1_Europe_24h.csv")

df = df.head()  # just to show an example

def to_country_code(lat: float, long: float) -> str:
    "Isn't this more generic than the country name as a string in the country's language?"
    return geolocator.reverse((lat, long)).raw['address']['country_code']
    
df['country_code'] = df[['latitude', 'longitude']].apply(
    lambda x: to_country_code(*x), axis=1
)
print(df)

   latitude  longitude  brightness  scan  track    acq_date  acq_time  \
0  44.48386   22.69465      310.45  2.20   1.44  2021-08-09       1.0   
1  42.92273   18.03089      306.02  3.97   1.85  2021-08-09       1.0   
2  42.91645   18.02446      307.58  3.97   1.85  2021-08-09       1.0   
3  41.26550   31.42957      302.74  1.01   1.01  2021-08-09       1.0   
4  42.02151   20.29289      300.76  2.87   1.62  2021-08-09       1.0   

  satellite  confidence version  bright_t31    frp daynight country_code  
0         A        80.0  6.1NRT      290.21  33.94        N           ro  
1         A        67.0  6.1NRT      293.46  42.59        N           ba  
2         A        72.0  6.1NRT      293.85  49.84        N           ba  
3         A        51.0  6.1NRT      292.64   5.38        N           tr  
4         A        33.0  6.1NRT      290.08  17.58        N           al

Before scaling up —since fire data shouldn't change too much— you could try caching the Nominatim responses.

This seems to work, after installing requests-cache:

requests_cache.install_cache('geopy_cache')
geolocator = Nominatim(
    user_agent="StackOverflow Question (https://stackoverflow.com/a/68727013/383793)"
)
requests_cache.uninstall_cache()

If you —rightfully so— feel iffy about monkeypatching. The Right Thing™ to do would be to pick up this issue in the geopy bugtracker.

user_3837930
Related