pandas-datareader World Bank API broken

Viewed 764

How come the latest example of the World Bank API doesn't work for pandas-datareader?

https://pandas-datareader.readthedocs.io/en/latest/remote_data.html#remote-data-wb

from pandas_datareader import wb

matches = wb.search('gdp.*capita.*const')
dat = wb.download(indicator='NY.GDP.PCAP.KD', country=['US', 'CA', 'MX'], start=2005, end=2008)
print(dat)

gives me this:

Traceback (most recent call last):
  File "C:\Python36\lib\site-packages\urllib3\connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "C:\Python36\lib\site-packages\urllib3\connectionpool.py", line 346, in _make_request
    self._validate_conn(conn)
  File "C:\Python36\lib\site-packages\urllib3\connectionpool.py", line 850, in _validate_conn
    conn.connect()
  File "C:\Python36\lib\site-packages\urllib3\connection.py", line 326, in connect
    ssl_context=context)
  File "C:\Python36\lib\site-packages\urllib3\util\ssl_.py", line 329, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "C:\Python36\lib\ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "C:\Python36\lib\ssl.py", line 814, in __init__
    self.do_handshake()
  File "C:\Python36\lib\ssl.py", line 1068, in do_handshake
    self._sslobj.do_handshake()
  File "C:\Python36\lib\ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Python36\lib\site-packages\requests\adapters.py", line 449, in send
    timeout=timeout
  File "C:\Python36\lib\site-packages\urllib3\connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "C:\Python36\lib\site-packages\urllib3\util\retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.worldbank.org', port=443): Max retries exceeded with url: /v2/indicators?per_page=50000&format=json (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/Jason/Google Drive/pycharm/test.py", line 3, in <module>
    matches = wb.search('gdp.*capita.*const')
  File "C:\Python36\lib\site-packages\pandas_datareader\wb.py", line 938, in search
    return WorldBankReader(**kwargs).search(string=string, field=field, case=case)
  File "C:\Python36\lib\site-packages\pandas_datareader\wb.py", line 809, in search
    indicators = self.get_indicators()
  File "C:\Python36\lib\site-packages\pandas_datareader\wb.py", line 745, in get_indicators
    resp = self._get_response(url)
  File "C:\Python36\lib\site-packages\pandas_datareader\base.py", line 155, in _get_response
    response = self.session.get(url, params=params, headers=headers)
  File "C:\Python36\lib\site-packages\requests\sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "C:\Python36\lib\site-packages\requests\sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Python36\lib\site-packages\requests\sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "C:\Python36\lib\site-packages\requests\adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='api.worldbank.org', port=443): Max retries exceeded with url: /v2/indicators?per_page=50000&format=json (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)'),))

It used to work on 0.7 version. I haven't ran it in a year or so, ran it today and gave me the same error so I upgraded to the latest datareader version and it's still not working.

2 Answers

I don't have a solution. But it was working on Friday. World Bank updated their certificate lately so that might be the reason. I am using World Bank data in my application so stuck here.

I have since raised a request in pandas_datareader github page: https://github.com/pydata/pandas-datareader/issues/791

I belive we have multiple issues here,

1- indicators suppose to be in array ['NY.GDP.PCAP.KD']

 dat = wb.download(indicator=['NY.GDP.PCAP.KD'], country=['US', 'CA', 'MX'], start=2005, end=2008)
    print(dat)

2- Now coming to the major issue, let's troubleshoot it and access your url directly on worldbank data, give try here, it is working, so the issue with pandas_datareader SSL (maybe socket client) that need update.

https://api.worldbank.org/v2/countries/US;CA;MX/indicators/NY.GDP.PCAP.KD?date=2005%3A2008&per_page=25000&format=json

3- Also in addition to the SSL issue, I faced another one related to some new limitation from world bank, to the size of data, I still need more investigation to confirm this point. Still, it is there (it may be related to their new certificate with URL length limitation, could be, check it below, please.

=== Update ===

it is confirmed by test, that there is number of countries limitation try to remove any country to reduce the number to 65 and it will work

https://api.worldbank.org/v2/countries/AFG;AGO;ARE;AUS;AUT;AZE;BEL;BGD;BHR;BRA;CAN;CHE;CHN;CZE;DEU;DNK;DZA;EGY;ESP;FIN;FRA;GBR;GHA;HKG;HUN;IDN;IND;IRL;IRN;IRQ;ITA;JOR;JPN;KAZ;KEN;KOR;KWT;LBN;LBY;LKA;MAR;MYS;NGA;NLD;OMN;PAK;PHL;POL;QAT;RUS;SAU;SDN;SGP;SOM;SWE;SYR;THA;TUR;TWN;TZA;UGA;UKR;USA;VNM;YEM;ZAF/indicators/AG.LND.ARBL.ZS?date=2005%3A2008&per_page=25000&format=json

===== Update 26-06-2020 ====

As for today the above link is working again, it seems they respond to my last week ticket.

enter image description here

Related