Is there a workaround to get past CoinGecko API security check?

user_4943299 0
Viewed 2209

I was running my code at work, and it just went smoothly, but on a different network (Home WiFi), I keep receiving 403 error when accessing CoinGecko V3 API. It can be observed that accessing the API on a private browser (to test it) will prompt a security check (hCaptcha) to be able to continue.

https://api.coingecko.com/api/v3/coins/bitcoin/market_chart?vs_currency=usd&days=30

with requests.get(url) as source:
    print(source.status_code)

I also tried pycoingecko (for checking) and still returns a similar error:

>>> from pycoingecko import CoinGeckoAPI
>>> cg = CoinGeckoAPI()
>>> cg.get_coin_market_chart_by_id(id='bitcoin', vs_currency='usd', days=1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\...\Python39\lib\site-packages\pycoingecko\utils.py", line 12, in input_args
    return func(*args, **kwargs)
  File "C:\Users\...\Python39\lib\site-packages\pycoingecko\api.py", line 169, in get_coin_market_chart_by_id
    return self.__request(api_url)
  File "C:\Users\...\Python39\lib\site-packages\pycoingecko\api.py", line 29, in __request
    response.raise_for_status()
  File "C:\Users\...\Python39\lib\site-packages\requests\models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://api.coingecko.com/api/v3/coins/bitcoin/market_chart?vs_currency=usd&days=1

I guess Cloudflare is suspecting my network activity, but of course I am using Python scripts to access the API, obviously. Is there a workaround to get past the security check?

hcaptcha coingecko cloudflare python
3 Answers

I can replicate a call to the endpoint on the command line with:

curl -vv -H "user-agent: C" https://api.coingecko.com/api/v3/coins/bitcoin/market_chart?vs_currency=USD&days=30

-vv - extra verbose logging - show detail on requests/response from the server

-H - set an HTTP header

With the user-agent: header set to blank I receive no response. If I set the value to any non-null value like C, Chrome, Chrome/91.0.0 it works.

If you can replicate the same in python you should get the same results... HTTP is a stateless protocol. If it doesn't work, something else is at play.

user_76190340

Just tried to do more than 50 requests per minute as said in CoinGecko docs and after approximately 200, I started to get error

urllib.error.HTTPError: HTTP Error 429: Too Many Requests

So the error you see is a result of particular activity.

In your case I guess the only way to through is to use a network proxy servers or to run parsing from some remote server like AWS EC2 instance, which is cheap and can work for parsing.

user_65176050

I think using scraperapi.com would be a good workaround for this. You get a few thousand requests to see if it works before having to pay and it's extremely easy to set up + designed to handle stuff like this where you might get a captcha/blocked etc.

Just tested it and works OK for me:

import requests
payload = {'api_key': 'YOUR API KEY',
           'url': 'https://api.coingecko.com/api/v3/coins/bitcoin/market_chart?vs_currency=usd&days=30'}
with requests.get('http://api.scraperapi.com', params=payload) as source:
    print(source.status_code)
user_134241850
Related